Download
minbpe is a minimal, clean implementation of byte-level Byte Pair Encoding (BPE), the tokenization approach widely used in modern language models. It operates on UTF-8 encoded bytes rather than Unicode characters, which makes it robust to arbitrary text inputs and avoids needing a language-specific character vocabulary. The repository is structured as a teaching-oriented implementation that shows how to train a tokenizer by learning merge rules, then apply those merges to encode text into token IDs and decode tokens back into text. It is intentionally small and readable so developers can understand each stage of BPE, including the mechanics of pair counting, merge application, and vocabulary growth. The project is especially useful for practitioners who want to demystify how LLM tokenizers work or who need a lightweight reference implementation for experimentation.
Features
- Byte-level BPE tokenizer implementation
- Tokenizer training via learned merge rules
- Encode and decode pipeline for text and token IDs
- UTF-8 byte handling for robust input coverage
- Readable minimal code for learning and experimentation
- Exercises and lecture-style materials for understanding BPE
Project Samples
