build nanoGPT
Video+code lecture on building nanoGPT from scratch
Build nanoGPT is an educational reproduction of GPT-style language-model training built step by step from an initially empty file. Its Git history is intentionally organized so learners can follow each architectural and training improvement as it is introduced. The accompanying video lecture explains how the code develops into a reproduction of the 124-million-parameter GPT-2 model. The project covers tokenization, transformer architecture, optimization, distributed training, data loading, and performance improvements. ...