Axolotl Files

Highlights

Sparse Finetuning using LLMCompressor

Using LLMCompressor, the integration allows users to efficiently fine-tune models with structured/unstructured sparsity, recovering 99% accuracy or better for sparse models, and 3X faster inference.

Quantization-Aware Training (QAT)

QAT simulates quantization during training to achieve higher quality post-training quantized (PTQ) models than from applying PTQ to models trained without QAT.

Mistral tokenizer support via mistral-common

Use Mistral's preferred mistral-common directly to support the preferred tokenization of chat messages.

Efficient chunked KD and online distillation

Use liger-style chunking to efficiently calculate KD-loss and add support for online distillation via logprobs from vllm/sglang. If you're

Miscellaneous

• Improved tool calling support
• Support for torch==2.5.1 will be deprecated in a future release. We recommend using torch 2.6.0 or 2.7.1.

What's Changed

• Add: Sparse Finetuning Integration with llmcompressor by @rahul-tuli in https://github.com/axolotl-ai-cloud/axolotl/pull/2479
• fix: remove doc string imports in monkeypatches by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2671
• Add ci and images for CUDA 12.8 for B200s by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2683
• Add num_proc to fix data set slow processing issue by @michelyang in https://github.com/axolotl-ai-cloud/axolotl/pull/2681
• Add missing init file to liger plugin by @BitPhinix in https://github.com/axolotl-ai-cloud/axolotl/pull/2670
• Make Axolotl Print Dataset Name Before Processing by @xzuyn in https://github.com/axolotl-ai-cloud/axolotl/pull/2668
• Fix: improve doc on merge/inference cli visibility by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2674
• Fix: Make MLflow config artifact logging respect hf_mlflow_log_artifa… by @C080 in https://github.com/axolotl-ai-cloud/axolotl/pull/2675
• Fix for setting adam_beta3 and adam_epsilon2 for CAME Optimizer by @xzuyn in https://github.com/axolotl-ai-cloud/axolotl/pull/2654
• GRPO fixes (peft) by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2676
• SP dataloader patching + removing custom sampler / dataloader logic by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2686
• feat(doc): clarify minimum pytorch and cuda to use blackwell by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2704
• fix: plugin rl overwriting trainer_cls by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2697
• feat: do not find turn indices if turn is not trainable by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2696
• SP context manager update by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2699
• Remove unused const by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2714
• models.py -> loaders/ module refactor by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2680
• update quarto for model loading refactor by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2716
• Allow Liger with GraniteMoE by @xzuyn in https://github.com/axolotl-ai-cloud/axolotl/pull/2715
• Fix quarto by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2717
• no need to generate diff file by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2728
• chore: update pre-commit hooks by @github-actions in https://github.com/axolotl-ai-cloud/axolotl/pull/2729
• Fix(doc): clarify data loading for local datasets and splitting samples by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2726
• feat(doc): note lora kernel incompat with RLHF by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2706
• Add chat templates for command-a and aya-23-8B models by @hyeobiiii in https://github.com/axolotl-ai-cloud/axolotl/pull/2731
• add two checks to handle legacy format interleaved multimodal ds by @sumo43 in https://github.com/axolotl-ai-cloud/axolotl/pull/2721
• Fix Mistral chat template (mistral_v7_tekken) by @mashdragon in https://github.com/axolotl-ai-cloud/axolotl/pull/2710
• feat(doc): add info on how to use dapo / dr grpo and misc doc fixes by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2673
• feat(doc): add google analytics to docs by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2708
• QAT by @SalmanMohammadi in https://github.com/axolotl-ai-cloud/axolotl/pull/2590
• Rank 0-only logging by @SalmanMohammadi in https://github.com/axolotl-ai-cloud/axolotl/pull/2608
• Lora kernels fix by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2732
• fix dist state init before deepspeed setup by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2737
• Add a few items to faq by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2734
• Fix: RL base feature parity by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2133
• fix(log): remove duplicate merge_lora param by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2742
• fix: suppress non-axolotl logs unless it's warning or higher by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2724
• add support for base image with uv by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2691
• chore: update pre-commit hooks by @github-actions in https://github.com/axolotl-ai-cloud/axolotl/pull/2745
• feat: add Group Relative Policy Optimization (GPRO) to RLHF documenta… by @mhenrichsen in https://github.com/axolotl-ai-cloud/axolotl/pull/2748
• remove deprecated wandb env var by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2751
• feat: add chat_template kwargs by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2694
• feat(modal): update docker tag to use torch2.6 from torch2.5 by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2749
• fix(deepspeed): deepspeed config not being set for z3 by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2754
• bump hf deps by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2735
• fix: remove hqq by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2759
• remove unused field for chat_template.default for DPO training by @timofey in https://github.com/axolotl-ai-cloud/axolotl/pull/2755
• add uv tooling for e2e gpu tests by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2750
• add manual seed for flaky test_geglu_backward test by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2763
• fix worker_init_fn signature handling by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2769
• handle when unable to save optimizer state when using ao optimizer with FSDP by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2773
• Fix the bug of position ids padding by @qywu in https://github.com/axolotl-ai-cloud/axolotl/pull/2739
• Feat: add tool calling support via tools column by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2774
• magistral small placeholder by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2777
• Data loader refactor by @djsaunde in https://github.com/axolotl-ai-cloud/axolotl/pull/2707
• build base images for torch 2.7.1 by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2764
• build 2.7.1 images and ci by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2784
• QAT docfix by @SalmanMohammadi in https://github.com/axolotl-ai-cloud/axolotl/pull/2778
• limit multipack sampler processes by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2771
• feat(doc): update readme to include changelog and remove matrix by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2775
• Fix logging import in evaluate.py (#2782) by @JZacaroli in https://github.com/axolotl-ai-cloud/axolotl/pull/2783
• update loss value for flakey e2e test by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2786
• Feat: Add Magistral and mistral-common tokenizer support by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2780
• support for QAT w RL (DPO) by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2776
• fix(doc): change grpo doc link by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2788
• Fix: adding magistral fsdp config, fixing not eval with test_datasets, handle mllama attention by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2789
• Fix: lora kernel pre-patch applied despite post-patch not applied by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2772
• fixed the lora_target_modules syntax inside examples/qwen2-vl/lora-7b.yaml by @cummins-orgs in https://github.com/axolotl-ai-cloud/axolotl/pull/2793
• KD fix w/ online distillation by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2700
• feat: remove evalfirst callback with built-in trainer arg by @NanoCode012 in https://github.com/axolotl-ai-cloud/axolotl/pull/2797
• release tag v0.10.0 by @winglian in https://github.com/axolotl-ai-cloud/axolotl/pull/2799

New Contributors

• @rahul-tuli made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2479
• @michelyang made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2681
• @C080 made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2675
• @github-actions made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2729
• @hyeobiiii made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2731
• @sumo43 made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2721
• @timofey made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2755
• @qywu made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2739
• @JZacaroli made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2783
• @cummins-orgs made their first contribution in https://github.com/axolotl-ai-cloud/axolotl/pull/2793

Full Changelog: https://github.com/axolotl-ai-cloud/axolotl/compare/v0.9.2...v0.10.0