Tiny LLM zh

Tiny LLM zh is an open-source educational project that demonstrates how to build and train compact Chinese large language models from scratch. It covers the complete language model development process, including tokenization, pretraining, supervised fine-tuning, and human preference alignment. The architecture uses Transformer components such as RMSNorm, rotary positional embeddings, and multi-head attention. It supports several model sizes, ranging from approximately 16 million to 440 million parameters. Training workflows use Hugging Face Transformers and DeepSpeed for distributed GPU training. The project includes datasets, training scripts, evaluation tools, and quantization resources. Additional implementations explore mixture-of-experts architectures and inference through vLLM and llama.cpp. Its primary purpose is education rather than production-grade model performance.