Transformer Engine v2.19 Release Notes

Key Features and Enhancements

• [Common] Added Rubin SM107a support when building with CUDA 13.4 or later. (#3275)
• [PyTorch] Added experimental per-direction hybrid quantization through CustomRecipe, allowing rowwise and columnwise paths to use different quantizers. (#2817)
• [Common, PyTorch] Added opt-in 2D block scaling for MXFP8 weights through MXFP8BlockScaling(enable_2d_quantization=True). (#2634)
• [Common, PyTorch] Added MXFP8 quantization support to NCCL-EP dispatch forward and combine backward, with GroupedLinear and fused grouped MLP support for prequantized grouped inputs. (#3270) (#3244)
• [Common, PyTorch] Added cuDNN FP8 fused attention support for packed THD inputs and context-parallel execution; FP8 THD sink-softmax backward requires cuDNN 9.26 or later. (#2994) (#3441)
• [PyTorch] Added experimental cuDNN-backed GatedDeltaNetAttention for dense and THD inputs. (#3351)
• [PyTorch] Enabled torch.compile(fullgraph=True) for supported FlashAttention and the non-FP8 unfused attention backend. (#3153) (#3189) (#3201) (#3286)
• [PyTorch] Added per-callable capture_time_hooks to make_graphed_callables for non-capturable warmup and graph-construction work. (#2831)
• [PyTorch] Added forward and backward support to FusedMLAQUpProjRopeQuant for fused MLA Q up-projection, RoPE, and MXFP8 quantization. (#3303) (#3330)
• [PyTorch] Expanded THD context parallelism with hierarchical a2a+p2p, experimental no-load-balance AllGather, and FlashAttention 4 on p2p, all_gather, and a2a paths. (#3290) (#3221) (#3438) (#3149)
• [PyTorch] Enabled row-scaled NVFP4 weight-gradient computation in dense Linear and GroupedLinear layers. (#3206) (#3324)
• [Common, PyTorch] Reduced MXFP8 grouped MLP CPU overhead. (#3338) (#3267) (#3238) (#3293)
• [PyTorch] Added named extra-output channels to te.ops.Sequential, allowing later fusible operations to consume outputs from earlier operations. (#3320)
• [PyTorch] Added newton_schulz_tp for tensor-parallel Newton-Schulz orthogonalization of replicated or sharded matrices. (#2920)
• [PyTorch] Optimized the performance and reduced parallel_cross_entropy memory use. Added overwrite_input option to reduce the memory usage even further by reusing the input storage. (#3273)
• [Common, PyTorch] Added NVFP4 stochastic-rounding and RHT split-quantization support on SM120 and SM121 GPUs. (#3281) (#3265)
• [JAX] Added per-head maximum-logit outputs to fused_attn and Flax DotProductAttention. (#3112)
• [JAX] Optimized experimental MoE Block to improve E2E perf, MXFP8 quantization, overflow detection, reduced other overheads (#3354)
• [JAX] Expert Parallelism APIs with reduced receive capacity and overflow detection (#3277)
• [JAX] Added ep_bootstrap support for meshes with axes orthogonal to expert parallelism by deriving communicator domains from the active mesh. (#3226)
• [Build] Added CMake discovery of NCCL headers installed by nvidia-nccl-cu12 and nvidia-nccl-cu13. (#3227)
• [Build] Enabled NVRTC to locate CUDA headers in Python site-packages at runtime, improving JIT compilation in relocated and build-isolated installations. (#3252)
• [Build] Reduced Transformer Engine library, wheel, and container sizes by compressing embedded CUDA binaries. (#3465)

Fixed Issues

• [Common] Disabled an affected cuBLAS 13.7 grouped GEMM algorithm on B300 and Rubin GPUs to prevent silent data corruption. (#3475)
• [PyTorch] Fixed FlashAttention 2 local and post-release validation, FlashAttention 3 sliding-window context-parallel checks, and FlashAttention 4 fallback for unsupported GPUs, head dimensions, or CUTLASS DSL configurations. (#3356) (#3341)
• [PyTorch] Fixed FusedAdam checkpoint step counters for parameter groups that are empty on some ranks. (#3318)
• [PyTorch] Fixed delayed-scaling FP8 activation recomputation when checkpointed callables enable FP8 in an inner autocast context. (#3284)
• [PyTorch] Fixed CUDA graph replay parameter-gradient lifetime by cloning returned parameter gradients by default, preventing later replays from overwriting retained gradients. (#2937)
• [PyTorch] Fixed a possible first-call segfault during lazy extension initialization by acquiring the GIL before Python imports. (#3255)
• [PyTorch] Fixed THD fused RoPE launch scaling for high sequence counts in token-linear forward and backward kernels. (#3057)
• [JAX] Fixed GEMM partition inference for tuple-valued mesh-axis specifications, including nested contracting axes and DP+FSDP layouts. (#3429) (#3272)
• [Build] Fixed NCCL-EP JIT in installed packages by shipping the runtime library and required JIT headers and discovering matching NCCL headers at runtime. (#3434)
• [Build] Fixed NCCL-EP source builds to respect MAX_JOBS and use a valid unlimited-parallel make invocation when no limit is set. (#3138) (#3243)

Breaking Changes in This Release

• [PyTorch] Changed mhc_fused_projection so its H output is always FP32 instead of matching the input dtype; cast it explicitly where BF16 is required. (#2978)
• [PyTorch] Changed the NCCL-EP Python API to support eager dispatch and drop-on-overflow: pass num_topk to ep_bootstrap, update positional ep_bootstrap and EpBuffer calls, and replace EpBuffer.token_counts with EpBuffer.tokens_per_expert. (#3229)

Deprecated Features

• [PyTorch] Deprecated the unused is_cg_capturable argument to parallel_cross_entropy; the operation is always CUDA graph capturable. (#3455)
• [PyTorch] Deprecated selecting GroupedLinear's grouped-tensor path with NVTE_GROUPED_LINEAR_USE_FUSED_GROUPED_GEMM; pass use_grouped_tensor=True or False to GroupedLinear instead. (#3224)
• [PyTorch] Deprecated implicit pointer-based layout detection when DotProductAttention receives Q/K/V views of packed buffers; pass the packed buffer through qkv_layer or kv_layer and set qkv_interleave_dim instead. (#3200)

Known Issues in This Release

• [PyTorch] GDN attention may generate a NaN in the backward pass when used with head_dim==32. The fix has been merged into nvidia-cudnn-frontend and will be resolved by the next release.