# BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes [BEFT](https://arxiv.org/abs/2509.15974) is a parameter efficient fine-tuning algorithm (PEFT) that only fine-tunes the added bias terms of value projections from pretrained transformer models. BEFT demonstrates that fine-tuning the added bias terms of value projections from pretrained transformers generally leads to a higher downstream performance in low-data regimes than fine-tuning the added bias terms of query/key projections. BEFT currently has the following tradeoffs: Pros: - BEFT requires far fewer parameters than LoRA, while maintaining competitive or superior performance across tasks in low-data regimes. Cons: - In high-data regimes, BEFT may show limited effectiveness compared to LoRA and full-parameters fine-tuning. If your use case belongs to the high-data regime, consider other PEFT methods such as LoRA. The abstract from the paper is: *Fine-tuning the bias terms of large language models (LLMs) has the potential to achieve unprecedented parameter efficiency while maintaining competitive performance, particularly in low-data regimes. However, the link between fine-tuning different bias terms (i.e., **b**_q, **b**_k, and **b**_v in the query, key, or value projections) and downstream performance remains largely unclear to date. In this paper, we investigate the link between fine-tuning **b**_q, **b**_k, and **b**_v with the performance of the downstream task. Our key finding is that directly fine-tuning **b**_v generally leads to higher downstream performance in low-data regimes, in comparison to **b**_q and **b**_k. We extensively evaluate this unique property across a wide range of LLMs spanning encoder-only and decoder-only architectures up to 6.7B parameters (including bias-free LLMs). Our results provide strong evidence for the effectiveness of directly fine-tuning **b**_v across various downstream tasks*. ## Benchmark overview # API ## BeftConfig [[autodoc]] tuners.beft.config.BeftConfig ## BeftModel [[autodoc]] tuners.beft.model.BeftModel