DeepSpeed Files

What's Changed

• Fix checkpoint rank selection for Ulysses sequence parallelism by @sfc-gh-thonguyen in https://github.com/deepspeedai/DeepSpeed/pull/8226
• Return a copy from OnebitLamb.get_lamb_coeffs by @vineethsaivs in https://github.com/deepspeedai/DeepSpeed/pull/8227
• Update version.txt after 0.19.5 release by @loadams in https://github.com/deepspeedai/DeepSpeed/pull/8243
• Implement the documented per-param-group lists in OneCycle by @vineethsaivs in https://github.com/deepspeedai/DeepSpeed/pull/8201
• Keep the elastic batch size within max_train_batch_size by @vineethsaivs in https://github.com/deepspeedai/DeepSpeed/pull/8237
• [DeepCompile] Fix KeyError on frozen parameters in ZeRO-3 by @roycho96 in https://github.com/deepspeedai/DeepSpeed/pull/8214
• [DeepCompile] Fix staticmethod handling on Python 3.9 by @baremetaldevx86 in https://github.com/deepspeedai/DeepSpeed/pull/8240
• Upgrade pip in Python install-smoke jobs by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8247
• (1/2) Implementing Compiler Pass for AutoTP by @therealnaveenkamal in https://github.com/deepspeedai/DeepSpeed/pull/8204
• Fix DeepCompile last use for unconsumed waits by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8254
• Add configurable sum gradient reduction by @zupengwang in https://github.com/deepspeedai/DeepSpeed/pull/8232
• Add native (DeepNVMe) host-memory pinning backend for accelerators by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8211
• Add compile.offload_activation_pin_memory for DeepCompile activation offload by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8258
• Fix int32 overflow in Triton grouped-GEMM expert offset by @PKUWZP in https://github.com/deepspeedai/DeepSpeed/pull/8261
• NCCL and other backend PG timeout 30min -> 10min by @stas00 in https://github.com/deepspeedai/DeepSpeed/pull/8253
• Fix ZeRO-3 crash in AutoTP universal-checkpoint metadata by @PKUWZP in https://github.com/deepspeedai/DeepSpeed/pull/8270
• Add CUDA graph support for HybridEngine generation by @PKUWZP in https://github.com/deepspeedai/DeepSpeed/pull/8271
• Fix docstring Args entries that name a parameter the function does not take by @darkdi in https://github.com/deepspeedai/DeepSpeed/pull/8223
• Keep parameter dtype through ZeRO-3 weight quantization by @adityasingh2400 in https://github.com/deepspeedai/DeepSpeed/pull/8215
• [AutoTP] Complete uneven sharding and universal checkpoint support by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8185
• Route FPDT and checkpoint writer pins through accelerator pin_memory by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8257
• Add fused triton kernel for swiglu by @hwchen2017 in https://github.com/deepspeedai/DeepSpeed/pull/8244
• Make pass contracts cover the passes DeepCompile actually schedules; Renaming files by @pengdurice in https://github.com/deepspeedai/DeepSpeed/pull/8251
• Non-reentrant activation checkpoint CPU offload by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8282
• Enable activation offloading by @pengdurice in https://github.com/deepspeedai/DeepSpeed/pull/8255
• [typo] Add raise for RuntimeError by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8281
• Fix ZeRO-1/2 with zero-sized parameters by @fwerkor in https://github.com/deepspeedai/DeepSpeed/pull/8280
• docs: note async cpu_checkpointing perf and expandable_segments by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8287
• Fix ZeroDivisionError in compute_elastic_config return_microbatch on non-0.2 elasticity by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8286
• Fix repeated FlopsProfiler metric accumulation by @baremetaldevx86 in https://github.com/deepspeedai/DeepSpeed/pull/8246
• fix: stop DeepSpeedConfig writing max_grad_norm back into the caller's config dict by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8289
• Enable DeepSpeed support on Apple Silicon (MPS) with ZeRO Stage 1-3 by @PKUWZP in https://github.com/deepspeedai/DeepSpeed/pull/8293
• fix(engine): resolve inference workspace attribute lookup by @nathon-lee in https://github.com/deepspeedai/DeepSpeed/pull/8288
• [AutoTP] Convert HF embedding_rowwise tp_plan entries to SKIP specs by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8294
• Fix zero reduce bucket size validation by @fwerkor in https://github.com/deepspeedai/DeepSpeed/pull/8266
• Register native pinned host memory with CUDA for GPU DMA by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8283
• Drop the documented grad_hooks ZeRO option, which does not exist by @vineethsaivs in https://github.com/deepspeedai/DeepSpeed/pull/8242
• Skip zero-sized parameters in HP fragment mapping by @fwerkor in https://github.com/deepspeedai/DeepSpeed/pull/8298
• perf(rollout): add HybridEngine rollout profiling by @nathon-lee in https://github.com/deepspeedai/DeepSpeed/pull/8295
• Route isend/irecv to nonblocking backend methods and stage them as async on MPS by @PKUWZP in https://github.com/deepspeedai/DeepSpeed/pull/8303
• Fix device mismatch in test_gate_up_partition_covers_the_whole_weight by @alanhuangyoo in https://github.com/deepspeedai/DeepSpeed/pull/8308
• Import print_dist in auto_tp by @alanhuangyoo in https://github.com/deepspeedai/DeepSpeed/pull/8311
• Reject invalid ZenFlow ratio and update interval boundaries by @tandede in https://github.com/deepspeedai/DeepSpeed/pull/8274
• Avoid redundant copies in MPS P2P staging by @FU-max-boop in https://github.com/deepspeedai/DeepSpeed/pull/8314
• Fix AutoTP metadata updates for unsharded modules by @zupengwang in https://github.com/deepspeedai/DeepSpeed/pull/8299
• [Apple Silicon Support Phase 1] Add Metal FusedAdam kernel and CPU Adam build for Apple Silicon by @PKUWZP in https://github.com/deepspeedai/DeepSpeed/pull/8300
• Enable optimized Adam backend for MuonWithAuxAdam optimizer by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8278
• feat(opsd): share prompt prefill across rollout samples by @nathon-lee in https://github.com/deepspeedai/DeepSpeed/pull/8296
• stop allocating per-element temporaries in the overflow check by @stas00 in https://github.com/deepspeedai/DeepSpeed/pull/8325
• Fix shared loss gradient accumulation by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8245
• Isolate DeepCompile list-schedule test ops by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8319
• Fix ZeRO-3 synchronization during OPSD rollout by @LiRunGuo in https://github.com/deepspeedai/DeepSpeed/pull/8264
• [AutoTP] Fix training lm_head routing by @gaoxiaomo in https://github.com/deepspeedai/DeepSpeed/pull/8302
• Route ZeRO/SuperOffload pin sites through accelerator pin_memory by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8256
• Honor adam_w_mode in the CPU multi_tensor_adam binding by @PKUWZP in https://github.com/deepspeedai/DeepSpeed/pull/8307

New Contributors

• @sfc-gh-thonguyen made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8226
• @baremetaldevx86 made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8240
• @darkdi made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8223
• @fwerkor made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8280
• @alanhuangyoo made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8308
• @tandede made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8274
• @FU-max-boop made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8314
• @LiRunGuo made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8264
• @gaoxiaomo made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8302

Full Changelog: https://github.com/deepspeedai/DeepSpeed/compare/v0.19.5...v0.19.6