FairScale
PyTorch extensions for high performance and large scale training
...The library also provides pipeline parallelism, activation checkpointing, mixed precision, optimizer state sharding (OSS), and auto-wrapping policies that reduce boilerplate in complex distributed setups. Its components are modular, so teams can adopt just the sharding optimizer or the pipeline engine without rewriting their training loop. FairScale puts emphasis on correctness and debuggability, offering hook points, logging, and reference examples for common trainer patterns. Although many ideas have since landed in core PyTorch, FairScale remains a valuable reference and a practical toolbox for squeezing more performance out of multi-GPU and multi-node jobs.