DeepSeed Files

What's Changed

• Validate warmup_type in WarmupCosineLR like WarmupLR by @sohumt123 in https://github.com/deepspeedai/DeepSpeed/pull/8151
• Run pull request code only inside a Modal Sandbox by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8170
• [AutoTP] Allow ZeRO stage 3 inference with tensor parallelism by @delock in https://github.com/deepspeedai/DeepSpeed/pull/8167
• Fix autotuning get_val_by_key to search all nested subdicts by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8177
• Enable support for Tutel when k != 1 for shared moe by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8174
• NVMe write warning by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8164
• Guard LRRangeTest and OneCycle schedulers against zero step sizes by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8166
• Fix WarmupLR collapsing multi-group base LRs to group 0's by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8171
• Stabilize fork-sensitive nightly tests and AutoSP coverage by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8172
• Make the OneCycle stair counts actually change the schedule by @vineethsaivs in https://github.com/deepspeedai/DeepSpeed/pull/8179
• [AutoTP] Enable HF colwise_gather_output to support lm_head replace by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8146
• Scope DeepCompile compiler state to graph and engine lifecycles by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8159
• Clarify merge commit exemption for signoff check in AGENTS.md and CLAUDE.md by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8194
• [AutoTP] Preserve universal checkpoint metadata for HuggingFace tp_plan by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8187
• Derive AutoEP rank splits from the per-expert count exchange by @yh0903 in https://github.com/deepspeedai/DeepSpeed/pull/8190
• Warn when zero.Init silently falls back to a single rank (#8084) by @akshansh47 in https://github.com/deepspeedai/DeepSpeed/pull/8089
• Release GIL in blocking AIO bindings by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8196
• Enable ZeRO-3 linear wrapper for existing models by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8189
• Skip CUDA transformer forward tests when TransformerBuilder is incompatible by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8200
• Clamp capacity to num_tokens in MoE gating drop branches (complete [#5353]) by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8155
• Preserve tensor learning rates across scheduler updates by @n33levo in https://github.com/deepspeedai/DeepSpeed/pull/8202
• [AutoTP] Fix ZeRO-3 checkpoint consolidation to gather across TP and DP by @delock in https://github.com/deepspeedai/DeepSpeed/pull/8168
• Add managed_gradient_accumulation for ZeRO stage 0/1 by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8184
• MoE dispatch: fix silent Tutel + TP corruption, speed up native and Tutel paths by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8195
• Add keyword argument support to activation checkpointing by @zupengwang in https://github.com/deepspeedai/DeepSpeed/pull/8182
• Remove redundant DCO workflow by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8188
• Preserve tensor learning rates in OneCycle by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8205
• Add Triton grouped-GEMM for MoE experts on Ampere/Ada by @hwchen2017 in https://github.com/deepspeedai/DeepSpeed/pull/8180
• Wiring up offload_opt_states by @pengdurice in https://github.com/deepspeedai/DeepSpeed/pull/8186
• Fix DeepCompile ZeRO-3 gathered parameter ownership by @tohtana in https://github.com/deepspeedai/DeepSpeed/pull/8157
• Consolidate LR scheduler tensor-LR tests into a contract suite by @delock in https://github.com/deepspeedai/DeepSpeed/pull/8206
• Fix MLU_Accelerator conformance to the DeepSpeedAccelerator ABC by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8208
• Reduce redundant work in AutoEP token routing by @hwchen2017 in https://github.com/deepspeedai/DeepSpeed/pull/8209
• Make PipelineModule.set_checkpoint_interval actually change the interval by @vineethsaivs in https://github.com/deepspeedai/DeepSpeed/pull/8178
• Share DeepNVMe pinned-tensor manager and route swap buffers through I/O handles by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8212
• Stop trim_mean sorting the caller's list in place by @vineethsaivs in https://github.com/deepspeedai/DeepSpeed/pull/8199
• Unmanaged gradient accumulation: ZeRO stage 2 support by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8203
• Fix ZeRO-3 all_reduce param fetch stride for padded parameters by @ebarkhordar in https://github.com/deepspeedai/DeepSpeed/pull/8158
• Unmanaged gradient accumulation: ZeRO stage 3 support by @sfc-gh-truwase in https://github.com/deepspeedai/DeepSpeed/pull/8217
• Update version.txt pre-0.19.4 release by @loadams in https://github.com/deepspeedai/DeepSpeed/pull/8218

New Contributors

• @sohumt123 made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8151
• @ebarkhordar made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8177
• @yh0903 made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8190
• @akshansh47 made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8089
• @n33levo made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8202
• @zupengwang made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8182
• @pengdurice made their first contribution in https://github.com/deepspeedai/DeepSpeed/pull/8186

Full Changelog: https://github.com/deepspeedai/DeepSpeed/compare/v0.19.3...v0.19.4