Search Results for "transformer"

Megatron-LM is a GPU-optimized deep learning framework from NVIDIA designed to train extremely large transformer-based language models efficiently at scale. The repository provides both a reference training implementation and Megatron Core, a composable library of high-performance building blocks for custom large-model pipelines. It supports advanced parallelism strategies including tensor, pipeline, data, expert, and context parallelism, enabling training across massive multi-GPU and multi-node clusters. ...

NanoGPT is a minimalistic yet powerful reimplementation of GPT-style transformers created by Andrej Karpathy for educational and research use. It distills the GPT architecture into a few hundred lines of Python code, making it far easier to understand than large, production-scale implementations. The repo is organized with a training pipeline (dataset preprocessing, model definition, optimizer, training loop) and inference script so you can train a small GPT on text datasets like Shakespeare...

...It is designed to accompany the video so students can inspect, modify, and rerun the implementation themselves. The repository begins with a simple character-level bigram language model and develops toward a GPT-style transformer. Its compact Python code exposes core ideas such as tokenization, embeddings, attention, transformer blocks, training, and text generation. The Git history mirrors much of the progression shown during the lecture. The implementation favors educational clarity over production optimization and includes known limitations in areas such as weight initialization. ...