Download
NeMo AutoModel is NVIDIA's open-source PyTorch Distributed training library for scaling LLM, VLM, diffusion, and retrieval-model training. Its DTensor-native SPMD approach lets the same training code scale from one GPU to large multi-node clusters by changing configuration. Hugging Face integration provides broad model compatibility without requiring format conversion. YAML recipes and CLI overrides keep experiments concise while preserving reproducibility. The library supports composable parallelism, optimized kernels, MoE acceleration, mixed precision, sequence packing, and asynchronous checkpointing. Jobs can run through interactive environments, Slurm, SkyPilot, or Kubernetes-based workflows. It targets both rapid research experiments and high-performance large-scale fine-tuning.
Features
- DTensor-native SPMD distributed training
- Hugging Face model compatibility
- YAML recipes with CLI overrides
- Composable tensor and pipeline parallelism
- MoE, mixed-precision, and kernel optimizations
- Slurm, SkyPilot, and Kubernetes execution
Project Samples
