huggingface--peft/docs/source/package_reference/unilora.md

UniLoRA

Uni-LoRA is a PEFT method that shares a compact trainable vector bank across low-rank adapter weights. Instead of learning every LoRA matrix element independently, UniLoRA deterministically projects entries into shared theta_d values and learns the shared parameters used by the adapter update.

Quick Start

from peft import UniLoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B")

config = UniLoraConfig(
    r=32,
    theta_d_length=256,
    proj_seed=42,
    target_modules=["q_proj", "v_proj"],
    unilora_dropout=0.0,
    init_weights=True,
    task_type="CAUSAL_LM",
)

peft_model = get_peft_model(model, config)
peft_model.print_trainable_parameters()

Important Parameters

r controls the low-rank adapter dimension. Larger values increase adapter capacity and memory use.

theta_d_length controls the length of the shared UniLoRA vector bank. This is the main trainable storage shared by the projected adapter entries.

proj_seed controls deterministic index generation for the fixed projections into theta_d. Reusing the same seed and configuration makes the generated adapter indices reproducible.

target_modules selects which modules receive UniLoRA adapters. Use module suffixes such as ["q_proj", "v_proj"], a regex string, or "all-linear" when supported by the model architecture.

unilora_dropout applies dropout inside UniLoRA adapter layers during training.

init_weights controls UniLoRA parameter initialization. Set it to False to keep a random theta_d initialization when you need to manage initialization manually.

save_indices controls whether UniLoRA checkpoints save the generated index and scale tensors together with the shared theta_d parameters. Keeping this disabled gives smaller checkpoints and regenerates indices from proj_seed; enabling it makes saved adapters independent from future index-generation changes.

Benchmark overview

API

UniLoraConfig

autodoc tuners.unilora.config.UniLoraConfig

UniLoraModel

autodoc tuners.unilora.model.UniLoraModel