Supported Fine-Tuning Strategies:
- Full Fine-Tuning: Adjusts all parameters of the model during training.
- Linear Probing: Fine-tunes only the residual blocks and the embedding layer, leaving other parameters unchanged.
- LoRA (Low-Rank Adaptation): A memory-efficient method that fine-tunes a small number of parameters by decomposing the weight matrices into low-rank matrices.
- DoRA (Directional LoRA): An extension of LoRA that decomposes pre-trained weights into magnitude and direction components. It uses LoRA for directional adaptation, enhancing learning capacity and stability without additional inference overhead.

Usage

Fine-Tuning Script

The provided finetune.py script allows you to fine-tune a model with specific configurations. You can customize various parameters to suit your dataset and desired fine-tuning strategy.

Example Usage:

source finetune.sh

This script runs the finetune.py file with a predefined set of hyperparameters for the model. You can adjust the parameters in the script as needed.

Available Options

Run the script with the --help flag to see a full list of available options and their descriptions:

python3 finetune.py --help

Script Configuration You can modify the following key parameters directly in the finetune.sh script: Fine-Tuning Strategy: Toggle between full fine-tuning, LoRA [--use-lora], DoRA [[--use-dora]], or Linear Probing [--use-linear-probing].

Performance Comparison

The figure below compares the performance of LoRA/DoRA against Linear Probing under the following conditions:

Training data split: 60% train, 20% validation, 20% test.
Benchmark: context_len=128, horizon_len=96
Fine-tuning: context_len=128, horizon_len=128
Black: Best result.
Blue: Second best result.