DeepSpeed
Deep learning optimization library: makes distributed training easy
...Achieve extreme compression for an unparalleled inference latency and model size reduction with low costs
DeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training landscape in terms of scale that is possible. These innovations such as ZeRO, 3D-Parallelism, DeepSpeed-MoE, ZeRO-Infinity, etc. fall under the training pillar.