Download Latest Version
NVIDIA Megatron Core 0.12.1 source code.tar.gz (6.4 MB)
Get an email when there's a new version of Megatron
| Name | Modified | Size | Downloads / Week |
|---|---|---|---|
| Parent folder | |||
| NVIDIA Megatron Core 0.9.0 source code.tar.gz | 2024-10-24 | 2.7 MB | |
| NVIDIA Megatron Core 0.9.0 source code.zip | 2024-10-24 | 3.3 MB | |
| README.md | 2024-10-24 | 703 Bytes | |
| Totals: 3 Items | 6.0 MB | 0 | |
- Uneven pipeline parallelism
- Enable pipeline parallelism where first and last ranks have fewer transformer layers than the intermediate ranks
- Per layer CUDAGraph support for GPT training with Transformer Engine modules
- Enable different TP sizes for the vision encoder
- Enable pipeline parallelism for T5 & Llava models
- Support multi-tile multi-image input in Llava models
- MoE
- FP8 support
- Runtime upcycling support
- Dispatcher implementation optimizations
- Shared expert support with overlapping optimizations
- Qwen Model support
- Known Issues
- When using sequence parallel, during the transformer block forward pass, dropout is not using the appropriate rng context.