FlashInfer Files

What's Changed

• feat: close feature gap by wiring up per-tensor routed FP8 fused-moe by @jdebache in https://github.com/flashinfer-ai/flashinfer/pull/4088
• Revert PR 4122 by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/4171
• [GDN] improve sm100 GDN performance by @Observer007 in https://github.com/flashinfer-ai/flashinfer/pull/4133
• fix(gdn): support WY decode on SM121 by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/4117
• fix(norm): convert float2 to e4m3 directly in packed cast by @elwhyjay in https://github.com/flashinfer-ai/flashinfer/pull/4167
• perf(test): bulk-precompile XQA decode kernels to cut test wall time ~4x by @bkryu in https://github.com/flashinfer-ai/flashinfer/pull/4119
• [perf] Optimize TRT-LLM routing for high-expert, high-top-k workloads by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/4152
• feat(xqa): ragged Q and per-row sliding-window masking for speculative decode by @yichengj0 in https://github.com/flashinfer-ai/flashinfer/pull/4137
• test(jit): assert BMM export symlink under GEN_SRC_DIR by @kahyunnam in https://github.com/flashinfer-ai/flashinfer/pull/4187
• Feat/ulysses p2p a2a by @forrestl111 in https://github.com/flashinfer-ai/flashinfer/pull/3820
• feat(moe_ep): MegaMoE framework integration ready: CUDA graph support, fused quant+stage launch, persistent knob cache, and prequantized weight packs by @mhoqueanik in https://github.com/flashinfer-ai/flashinfer/pull/4079
• docs: document CuTe prefill scheduling override by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/4162
• docs: add missing trtllm_fp8_per_tensor_scale_routed_moe API entry by @kangbintNV in https://github.com/flashinfer-ai/flashinfer/pull/4175
• docs(mamba): document checkpointing varlen arguments by @hebo1221 in https://github.com/flashinfer-ai/flashinfer/pull/4129
• Yanqinz/fix-gemm-and-grouped-mm-test-issue by @yanqinz2 in https://github.com/flashinfer-ai/flashinfer/pull/4185
• feat(comm): extend trtllm_allreduce to SM12x and fix lamport buffer pointer packing by @yichengj0 in https://github.com/flashinfer-ai/flashinfer/pull/3903
• feat(moe): add unified unpacked pre-routed FP4 mode by @feih-nv in https://github.com/flashinfer-ai/flashinfer/pull/4104
• feat(mla): support packed low-head and variable-Q decode by @PerkzZheng in https://github.com/flashinfer-ai/flashinfer/pull/4178
• bump version to 0.6.16 by @jimmyzho in https://github.com/flashinfer-ai/flashinfer/pull/4142
• [fix]fix xqa flaky test on spark by @qsang-nv in https://github.com/flashinfer-ai/flashinfer/pull/4161
• fix: make mxfp8 gemm test pass by having it quantize along the correct dimension by @jdebache in https://github.com/flashinfer-ai/flashinfer/pull/3882
• fix(moe): serialize CuTe DSL autotune replay by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/4192
• feat(moe_ep): fault-tolerance rank mask (NCCL-EP + NIXL-EP) by @Anerudhan in https://github.com/flashinfer-ai/flashinfer/pull/4183
• fix(xqa): fix PDL load ordering and SM90 fp8 draft-mask dispatch by @yichengj0 in https://github.com/flashinfer-ai/flashinfer/pull/4199
• feat(msa): accept K/V views split from a packed paged KV cache by @yichengj0 in https://github.com/flashinfer-ai/flashinfer/pull/4039
• feat(moe): add TRTLLM MXFP4 W4A8 and W4A16 unified API support by @feih-nv in https://github.com/flashinfer-ai/flashinfer/pull/4159
• [cli] add CLI helper for flashinfer-jit-cache and flashinfer-cubin wheel installs by @dierksen in https://github.com/flashinfer-ai/flashinfer/pull/3142
• [feat] Add SITU trtllmgen MOE by @jiahanc in https://github.com/flashinfer-ai/flashinfer/pull/4180
• feat(sm120): fused MoE (SwiGLU) via moe_gemm is_gated for cute SM120 groupwise GEMM by @CarstyYou in https://github.com/flashinfer-ai/flashinfer/pull/4130
• fix(moe): pad trtllm-gen route map by one element to avoid OOB read by @syuoni in https://github.com/flashinfer-ai/flashinfer/pull/4237
• perf(moe_ep): CuTe-DSL 4.5.2 mainloop WAR — drop the 4.6.1 runtime floor by @mhoqueanik in https://github.com/flashinfer-ai/flashinfer/pull/4101
• Fix the routing inconsistency for num_groups > 1 by @b8zhong in https://github.com/flashinfer-ai/flashinfer/pull/3946
• fix: support host global scale in CuTe-DSL NVFP4 quantization by @akurathiswaraj in https://github.com/flashinfer-ai/flashinfer/pull/4138
• fix/test(moe_ep): self-bootstrap 1-rank process group in dg mega oracle test by @mhoqueanik in https://github.com/flashinfer-ai/flashinfer/pull/4221
• feat: support native qk_rope_head_dim=0 sparse MLA decode in trtllm-gen by @JustinTong0323 in https://github.com/flashinfer-ai/flashinfer/pull/4108
• feat(topk): support separate page table row starts by @zianglih in https://github.com/flashinfer-ai/flashinfer/pull/4169
• feat(comm): make mixed-comm VMM workspaces checkpointable by @galletas1712 in https://github.com/flashinfer-ai/flashinfer/pull/3910
• SM 107 Reland + Merge Back from v0.6.16 Release Branch by @Vinnie6167 in https://github.com/flashinfer-ai/flashinfer/pull/4280
• test(sm103): fix FP4 autotuner cache inspection by @tiffany940107 in https://github.com/flashinfer-ai/flashinfer/pull/4145
• bump version to 0.6.17 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/4283
• feat(gemm): sync mm_fp4 SM120 NVFP4 dense GEMM kernel to b12x HEAD by @yichengj0 in https://github.com/flashinfer-ai/flashinfer/pull/4253
• fix(b12x): correct fp4 quantization numerics and add input_global_scale to decouple weight and activation scales by @yichengj0 in https://github.com/flashinfer-ai/flashinfer/pull/3932
• feat(moe): support shared expert fusion for trtllm-gen fp4 moe by @Aneureka in https://github.com/flashinfer-ai/flashinfer/pull/4239
• fix(test): repair MoEFinalizeConfig call site in b12x unified MoE tests (#4395) by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/4408
• revert(moe): revert [#3738] SM90 CUTLASS MoE backend (+ dependents [#4025], [#4080]) on release-v0.6.17 by @aleozlx in https://github.com/flashinfer-ai/flashinfer/pull/4411

New Contributors

• @hebo1221 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/4129
• @akurathiswaraj made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/4138
• @JustinTong0323 made their first contribution in https://github.com/flashinfer-ai/flashinfer/pull/4108

Full Changelog: https://github.com/flashinfer-ai/flashinfer/compare/v0.6.16rc5...v0.6.17