Search Results for "transformer"

Megatron-LM is a GPU-optimized deep learning framework from NVIDIA designed to train extremely large transformer-based language models efficiently at scale. The repository provides both a reference training implementation and Megatron Core, a composable library of high-performance building blocks for custom large-model pipelines. It supports advanced parallelism strategies including tensor, pipeline, data, expert, and context parallelism, enabling training across massive multi-GPU and multi-node clusters. ...

LLM101n is an educational repository that walks you through building and understanding large language models from first principles. It emphasizes intuition and hands-on implementation, guiding you from tokenization and embeddings to attention, transformer blocks, and sampling. The materials favor compact, readable code and incremental steps, so learners can verify each concept before moving on. You’ll see how data pipelines, batching, masking, and positional encodings fit together to train a small GPT-style model end to end. The repo often complements explanations with runnable notebooks or scripts, encouraging experimentation and modification. ...

...Its notebooks are written to make the mechanics visible, so learners can understand gradients, backpropagation, embeddings, multilayer perceptrons, batch normalization, and Transformer-style ideas through code. The course is especially valuable because it avoids treating neural networks as black boxes. nn-zero-to-hero is best suited for programmers who want a practical, code-first introduction to how modern neural networks work.

...It is designed to accompany the video so students can inspect, modify, and rerun the implementation themselves. The repository begins with a simple character-level bigram language model and develops toward a GPT-style transformer. Its compact Python code exposes core ideas such as tokenization, embeddings, attention, transformer blocks, training, and text generation. The Git history mirrors much of the progression shown during the lecture. The implementation favors educational clarity over production optimization and includes known limitations in areas such as weight initialization. ...