Halo

Halo is an open-source training framework for large language and multimodal models, designed to scale from a single GPU to multi-node clusters. It supports pretraining, supervised fine-tuning, preference optimization, distillation, and fully asynchronous multi-turn reinforcement learning. The framework trains Hugging Face models directly, so checkpoints remain compatible with standard from_pretrained workflows. Distributed execution supports expert, context, tensor, and expert-tensor parallelism alongside FSDP2. Performance features include FlashAttention, DeepEP, Liger kernels, Grouped GEMM, BF16 optimizers, LoRA, and QLoRA. RL rollouts can run through vLLM or SGLang while overlapping with training. Halo also provides Docker images, YAML recipes, checkpoint tools, dataset utilities, benchmarks, and agent-oriented technical documentation.