Search Results for "high performance computing"

Megatron-LM is a GPU-optimized deep learning framework from NVIDIA designed to train extremely large transformer-based language models efficiently at scale. The repository provides both a reference training implementation and Megatron Core, a composable library of high-performance building blocks for custom large-model pipelines. It supports advanced parallelism strategies including tensor, pipeline, data, expert, and context parallelism, enabling training across massive multi-GPU and multi-node clusters. The framework includes mixed-precision training options such as FP16, BF16, FP8, and FP4 to maximize performance and memory efficiency on modern hardware. ...

The Library is for Finite Transforms such as the Number Theoretic Transform (NTT) and Finite Radon Transform (FRT). Current modules include NTTW for NTTs with high resolution (microsecond) timing, basic array and imaging. The transforms are optimised for performance.