Search Results for "high performance computing"

Megatron-LM is a GPU-optimized deep learning framework from NVIDIA designed to train extremely large transformer-based language models efficiently at scale. The repository provides both a reference training implementation and Megatron Core, a composable library of high-performance building blocks for custom large-model pipelines. It supports advanced parallelism strategies including tensor, pipeline, data, expert, and context parallelism, enabling training across massive multi-GPU and multi-node clusters. The framework includes mixed-precision training options such as FP16, BF16, FP8, and FP4 to maximize performance and memory efficiency on modern hardware. ...

...Using TensorFlow, it defines a meta-optimizer model that learns by observing and adapting to the optimization trajectories of other models. The project allows users to compare performance between traditional optimizers and the learned optimizer (L2L) on various benchmarks, demonstrating how optimization strategies can be learned through experience. The design supports both single-variable and high-dimensional problems, and includes tools for evaluating how well a learned optimizer performs on unseen tasks.