Here's a Colab notebook for training a feedback transformer on Tiny Shakespeare dataset.
\n\n": "Here's a Colab notebook for training a feedback transformer on Tiny Shakespeare dataset.
\n\n", "The default configs can and will be over-ridden when we start the experiment
\n": "The default configs can and will be over-ridden when we start the experiment
\n", "It will download from the url if not present
\n": "It will download from the url if not present
\n", "\n": "
\n", "\n": "\n", "
_^_0_^_ has shape _^_1_^_
\n": "_^_0_^_ has shape _^_1_^_
\n", "Call the model, with the all but the last token
\n": "Call the model, with the all but the last token
\n", "Compute gradients
\n": "Compute gradients
\n", "Cross entropy loss
\n": "Cross entropy loss
\n", "Dataloader
\n": "Dataloader
\n", "Dataset
\n": "Dataset
\n", "GPT-2 configs
\n": "GPT-2 configs
\n", "GPT-2 hugging face uses 1D Convolution layers. We need to transpose those weights since we use linear layers
\n": "GPT-2 hugging face uses 1D Convolution layers. We need to transpose those weights since we use linear layers
\n", "Get cross entropy loss
\n": "Get cross entropy loss
\n", "Huggingface tokenizer
\n": "Huggingface tokenizer
\n", "Initialize the GPT2 model
\n": "Initialize the GPT2 model
\n", "Initialize the data loader
\n": "Initialize the data loader
\n", "Initialize the optimizer
\n": "Initialize the optimizer
\n", "LoRA rank
\n": "LoRA rank
\n", "Load out model. We use _^_0_^_ because the state does not have LoRA weights
\n": "Load out model. We use _^_0_^_ because the state does not have LoRA weights
\n", "Load pre-trained model weights
\n": "Load pre-trained model weights
\n", "Load the huggingface model and get the parameters
\n": "Load the huggingface model and get the parameters
\n", "Log the loss
\n": "Log the loss
\n", "Make gradients 0
\n": "Make gradients 0
\n", "Mapping (_^_0_^_) of decoder layers
\n": "Mapping (_^_0_^_) of decoder layers
\n", "Move _^_0_^_ to device
\n": "Move _^_0_^_ to device
\n", "Move the parameters based on mapping
\n": "Move the parameters based on mapping
\n", "Optimize
\n": "Optimize
\n", "Optimizer
\n": "Optimizer
\n", "Training configs
\n": "Training configs
\n", "Transformer embedding and prediction layer parameter mapping (_^_0_^_)
\n": "Transformer embedding and prediction layer parameter mapping (_^_0_^_)
\n", "make sure that only lora weights are not loaded
\n": "make sure that only lora weights are not loaded
\n", "Finetune GPT-2 with LoRA": "Finetune GPT-2 with LoRA", "This is training code with notes for fine-tuning pre-trained GPT-2 model with LoRA.": "This is training code with notes for fine-tuning pre-trained GPT-2 model with LoRA." }