Download Latest Version v0.21.0 source code.zip (12.2 MB) Google Add to Preferred Sources
Home / v0.21.0
Name Modified Size InfoDownloads / Week
Parent folder
README.md 2026-09-15 21.3 kB
0
v0.21.0 source code.tar.gz 2026-09-15 11.2 MB
1 1 weekly downloads
v0.21.0 source code.zip 2026-09-15 12.2 MB
0
Totals: 3 Items   23.5 MB 1

Highlights

peft-v0 21 0

New methods

Riemannian-preconditional LoRA optimizer

Riemannian-preconditioned LoRA optimizer (#3382), contributed by @smellslikeml, is a new LoRA-specific optimizer similar to LoRA-FA or LoRA+ that applies a Riemannian pre-conditioning on the gradients to improve feature learning and avoiding separate tuning of learning rates for LoRA's A and B matrices.

LoRA variant: Knowledge-Aware Singular-Value Adaption (KaSA)

KaSA aims at identifying knowledge aware singular values from the base weight. It then removes noisy knowledge from those weights and specifically trains the singular values most relevant to the task. Implemented in [#3446] by @robbiebusinessacc and @iambogeumkim.

Super-Tuning (#3518)

Super-Tuning uses calibration-free magnitude scoring to allocate a sparse tuning budget. With BottomK selection, Super adapts low-magnitude weights, motivated by the idea that they provide degrees of freedom for learning a new task while leaving larger-magnitude pretrained weights untouched. Supra adds LoRA on top of this sparse update. It is a good fit for exploring sparsity alongside higher-rank adaptation, particularly on reasoning-heavy fine-tuning tasks. Thanks to @smellslikeml.

ShadowPEFT (#3354)

ShadowPEFT runs a small shadow network alongside the frozen base model. That shadow state gets updated at every decoder layer, so adaptation is shared across the stack instead of a separate low-rank update on each targeted linear layer (like LoRA). It’s a good choice when you care about downstream or OOD performance and want to stay in the same trainable-parameter range as LoRA, or when you want the small shadow on the edge and the frozen base in the cloud. This works best if the shadow model starts from a small, pretrained model. Contributed by @SeanLee97.

Enhancements

  • Speed up MoE target parameter computation (#3577): For the common use case of one active adapter, weight and MoE LoRA delta computation is folded into a single baddbmm operation. Expect slight numerical differences in this case.

  • Support torchao v0.18.0 by supporting the newer Int8Tensor, Float8Tensor, etc. tensor subclasses.

  • Orthogonal Subspace Fine-tuning received a significant update in [#3524] reducing the memory footprint by 22% and the training time by 46% according to the MetaMathQA benchmark while keeping task accuracy and forgetting stable. Expect slight numerical differences, especially in fp16, bf16.

  • https://github.com/huggingface/peft/pull/3639: Thanks to @amogh-nagri-11, prompt learning methods now allow to delete an adapter. For adaption prompt, adding or loading a second adapter no longer automatically activates it, in line with how other PEFT methods handle this.

Changes

Refactored state dict handling (#3490)

An internal change moved state dict loading/saving method-specific code paths to the respective tuners. This should be transparent to the end-user and was extensively tested and should work exactly the same as previously, but if you suspect that the model behaves differently, let us know.

Preserved adapter trainablity during injection (#3495)

Previously, when loading or adding a second, third etc. adapter, this could affect the requires_grad of the existing adapter (or base model), depending on the value of is_trainable. This should no longer be the case thanks to @DaoyuanLi2816.

Bug fixes for LoRA adapter combining (#3111)

Combining LoRA adapters using add_weighted_adapter had a bug in the SVD path which led to the LoRA scaling being applied twice. This is now fixed.

OFT uses generic quantization backend (#3355)

We moved OFT from a method-specific quantization backend to the generic quantization backend. This affects bitsandbytes, GPT-Q, etc. This should not affect existing checkpoints, but please let us know if you run into issues.

Stricter checks on adapter handling

  • [#3667]: inject_adapter_in_model now rejects PEFT methods that have shared state that would live on the parent model (like VeRA), as no such parent model exists. Contributed by @CoralGarden52.

  • [#3534]: Thanks to @LK-maker-007, the PEFT now throws an error when adding a second adapter to the PEFT model using add_adapter that targets non-existing layers. Before, the adapter would just not adapt anything but be silent about it. peft_model.targeted_module_names and peft_model.targeted_parameter_names no longer contain duplicates. They are still sorted by insertion order (not sets). To suppress potential errors, catch peft.PeftError.

  • [#3504]: When deleting an adapter, there is now a check if the adapter is merged. If it's merged, an error is raised, since deleting it would not remove its effect. This was contributed by @DaoyuanLi2816.

All Changes

New Contributors

Full Changelog: https://github.com/huggingface/peft/compare/v0.20.0...v0.21.0

Source: README.md, updated 2026-09-15