chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1>Low-Rank Adaptation (LoRA)</h1>\n<p>This is an implementation of <a href=\"https://arxiv.org/abs/2106.09685\">Low-Rank Adaptation (LoRA)</a> in <a href=\"https://pytorch.org\">PyTorch</a>.</p>\n<p>Low-Rank Adaptation (LoRA) freezes pre-trained model weights and injects trainable rank decomposition matrices into each layer of the transformer. This makes it possible to efficiently fine-tune large langauge models by reducing trainable parameters by a large factor.</p>\n<p>Here's <a href=\"experiment.html\">the training code</a> for training a GPT2 model with LoRA on Tiny Shakespeare dataset.</p>\n": "<h1>Low-Rank Adaptation (LoRA)</h1>\n<p>This is an implementation of <a href=\"https://arxiv.org/abs/2106.09685\">Low-Rank Adaptation (LoRA)</a> in <a href=\"https://pytorch.org\">PyTorch</a>.</p>\n<p>Low-Rank Adaptation (LoRA) freezes pre-trained model weights and injects trainable rank decomposition matrices into each layer of the transformer. This makes it possible to efficiently fine-tune large langauge models by reducing trainable parameters by a large factor.</p>\n<p>Here's <a href=\"experiment.html\">the training code</a> for training a GPT2 model with LoRA on Tiny Shakespeare dataset.</p>\n",
|
||||
"<h2>LoRA Embedding Layer</h2>\n<p>Similar to LoRA linear layer this adds a low-rank decomposition to the pre-trained embedding weights matrix (<span translate=no>_^_0_^_</span>).</p>\n<p><span translate=no>_^_1_^_</span></p>\n": "<h2>LoRA Embedding Layer</h2>\n<p>Similar to LoRA linear layer this adds a low-rank decomposition to the pre-trained embedding weights matrix (<span translate=no>_^_0_^_</span>).</p>\n<p><span translate=no>_^_1_^_</span></p>\n",
|
||||
"<h2>LoRA Linear Layer</h2>\n<p>LoRA linear layer adds a low-rank decomposition to the pre-trained weight matrix (<span translate=no>_^_0_^_</span>) of the linear layer.</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>, where <span translate=no>_^_2_^_</span>, <span translate=no>_^_3_^_</span>, and the rank <span translate=no>_^_4_^_</span>.</p>\n<p>All parameters are frozen except <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>.</p>\n<p><span translate=no>_^_7_^_</span> is initialized to be zero at the beginning of the training.</p>\n<p>They multiple <span translate=no>_^_8_^_</span> by <span translate=no>_^_9_^_</span> where <span translate=no>_^_10_^_</span> is a hyper-parameter. Once <span translate=no>_^_11_^_</span> is tuned it can be kept the same when varying <span translate=no>_^_12_^_</span>.</p>\n": "<h2>LoRA Linear Layer</h2>\n<p>LoRA linear layer adds a low-rank decomposition to the pre-trained weight matrix (<span translate=no>_^_0_^_</span>) of the linear layer.</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>, where <span translate=no>_^_2_^_</span>, <span translate=no>_^_3_^_</span>, and the rank <span translate=no>_^_4_^_</span>.</p>\n<p>All parameters are frozen except <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>.</p>\n<p><span translate=no>_^_7_^_</span> is initialized to be zero at the beginning of the training.</p>\n<p>They multiple <span translate=no>_^_8_^_</span> by <span translate=no>_^_9_^_</span> where <span translate=no>_^_10_^_</span> is a hyper-parameter. Once <span translate=no>_^_11_^_</span> is tuned it can be kept the same when varying <span translate=no>_^_12_^_</span>.</p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Add <span translate=no>_^_0_^_</span> </p>\n": "<p>Add <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Bias parameter <span translate=no>_^_0_^_</span> (also frozen) </p>\n": "<p>Bias parameter <span translate=no>_^_0_^_</span> (also frozen) </p>\n",
|
||||
"<p>Compute <span translate=no>_^_0_^_</span> </p>\n": "<p>Compute <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Compute the embeddings <span translate=no>_^_0_^_</span> </p>\n": "<p>Compute the embeddings <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Freeze it </p>\n": "<p>Freeze it </p>\n",
|
||||
"<p>Initialize <span translate=no>_^_0_^_</span> similar to a weight matrix in a normal linear layer </p>\n": "<p>Initialize <span translate=no>_^_0_^_</span> similar to a weight matrix in a normal linear layer </p>\n",
|
||||
"<p>Initialize <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> so that <span translate=no>_^_2_^_</span> is <span translate=no>_^_3_^_</span> at initialization </p>\n": "<p>Initialize <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> so that <span translate=no>_^_2_^_</span> is <span translate=no>_^_3_^_</span> at initialization </p>\n",
|
||||
"<p>Initialize <span translate=no>_^_0_^_</span> with a normal distribution </p>\n": "<p>Initialize <span translate=no>_^_0_^_</span> with a normal distribution </p>\n",
|
||||
"<p>Matrix <span translate=no>_^_0_^_</span> </p>\n": "<p>Matrix <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Matrix <span translate=no>_^_0_^_</span>, we keep <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> transposed </p>\n": "<p>Matrix <span translate=no>_^_0_^_</span>, we keep <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> transposed </p>\n",
|
||||
"<p>No bias parameter </p>\n": "<p>No bias parameter </p>\n",
|
||||
"<p>Set <span translate=no>_^_0_^_</span> is not provided. i.e. make the scaling factor <span translate=no>_^_1_^_</span>. </p>\n": "<p>Set <span translate=no>_^_0_^_</span> is not provided. i.e. make the scaling factor <span translate=no>_^_1_^_</span>. </p>\n",
|
||||
"<p>The pre-trained embedding weights <span translate=no>_^_0_^_</span> (frozen) </p>\n": "<p>The pre-trained embedding weights <span translate=no>_^_0_^_</span> (frozen) </p>\n",
|
||||
"<p>The pre-trained weight <span translate=no>_^_0_^_</span> </p>\n": "<p>The pre-trained weight <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>scaling factor <span translate=no>_^_0_^_</span> </p>\n": "<p>scaling factor <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number embedding dimensions </li>\n<li><span translate=no>_^_2_^_</span> is the rank of the decomposition <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> is the scaling factor <span translate=no>_^_5_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> is the number of embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number embedding dimensions </li>\n<li><span translate=no>_^_2_^_</span> is the rank of the decomposition <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> is the scaling factor <span translate=no>_^_5_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of input features of the linear layer </li>\n<li><span translate=no>_^_1_^_</span> is the number of output features of the linear layer </li>\n<li><span translate=no>_^_2_^_</span> is a flag indicating if there is a bias parameter </li>\n<li><span translate=no>_^_3_^_</span> is the rank of the decomposition <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the scaling factor <span translate=no>_^_6_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> is the number of input features of the linear layer </li>\n<li><span translate=no>_^_1_^_</span> is the number of output features of the linear layer </li>\n<li><span translate=no>_^_2_^_</span> is a flag indicating if there is a bias parameter </li>\n<li><span translate=no>_^_3_^_</span> is the rank of the decomposition <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the scaling factor <span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"Annotated implementation of RoRA from paper LoRA: Low-Rank Adaptation of Large Language Models": "Annotated implementation of RoRA from paper LoRA: Low-Rank Adaptation of Large Language Models",
|
||||
"Low-Rank Adaptation (LoRA)": "Low-Rank Adaptation (LoRA)"
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"<h1>Finetune <a href=\"gpt2.html\">GPT-2</a> with <a href=\"index.html\">LoRA</a></h1>\n<p>Here's a Colab notebook for training a feedback transformer on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/lora/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>Finetune <a href=\"gpt2.html\">GPT-2</a> with <a href=\"index.html\">LoRA</a></h1>\n<p>Here's a Colab notebook for training a feedback transformer on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/lora/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Trainer configurations and the training loop</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>Trainer configurations and the training loop</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n",
|
||||
"<h3>Initialize the model, optimizer and dataloader</h3>\n": "<h3>Initialize the model, optimizer and dataloader</h3>\n",
|
||||
"<h3>Load pre-trained <a href=\"https://huggingface.co/openai-community/gpt2\">GPT-2 from huggingface</a></h3>\n": "<h3>Load pre-trained <a href=\"https://huggingface.co/openai-community/gpt2\">GPT-2 from huggingface</a></h3>\n",
|
||||
"<h3>Tiny Shakespeare dataset</h3>\n<p>It will download from the url if not present</p>\n": "<h3>Tiny Shakespeare dataset</h3>\n<p>It will download from the url if not present</p>\n",
|
||||
"<h3>Training loop</h3>\n": "<h3>Training loop</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p><a href=\"gpt2.html\">GPT2 model</a> </p>\n": "<p><a href=\"gpt2.html\">GPT2 model</a> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Call the model, with the all but the last token </p>\n": "<p>Call the model, with the all but the last token </p>\n",
|
||||
"<p>Compute gradients </p>\n": "<p>Compute gradients </p>\n",
|
||||
"<p>Cross entropy loss </p>\n": "<p>Cross entropy loss </p>\n",
|
||||
"<p>Dataloader </p>\n": "<p>Dataloader </p>\n",
|
||||
"<p>Dataset </p>\n": "<p>Dataset </p>\n",
|
||||
"<p>GPT-2 configs </p>\n": "<p>GPT-2 configs </p>\n",
|
||||
"<p>GPT-2 hugging face uses 1D Convolution layers. We need to transpose those weights since we use linear layers </p>\n": "<p>GPT-2 hugging face uses 1D Convolution layers. We need to transpose those weights since we use linear layers </p>\n",
|
||||
"<p>Get cross entropy loss </p>\n": "<p>Get cross entropy loss </p>\n",
|
||||
"<p>Huggingface tokenizer </p>\n": "<p>Huggingface tokenizer </p>\n",
|
||||
"<p>Initialize the <a href=\"gpt2.html\">GPT2 model</a> </p>\n": "<p>Initialize the <a href=\"gpt2.html\">GPT2 model</a> </p>\n",
|
||||
"<p>Initialize the data loader </p>\n": "<p>Initialize the data loader </p>\n",
|
||||
"<p>Initialize the optimizer </p>\n": "<p>Initialize the optimizer </p>\n",
|
||||
"<p>LoRA rank </p>\n": "<p>LoRA rank </p>\n",
|
||||
"<p>Load out model. We use <span translate=no>_^_0_^_</span> because the state does not have LoRA weights </p>\n": "<p>Load out model. We use <span translate=no>_^_0_^_</span> because the state does not have LoRA weights </p>\n",
|
||||
"<p>Load pre-trained model weights </p>\n": "<p>Load pre-trained model weights </p>\n",
|
||||
"<p>Load the huggingface model and get the parameters </p>\n": "<p>Load the huggingface model and get the parameters </p>\n",
|
||||
"<p>Log the loss </p>\n": "<p>Log the loss </p>\n",
|
||||
"<p>Make gradients 0 </p>\n": "<p>Make gradients 0 </p>\n",
|
||||
"<p>Mapping (<span translate=no>_^_0_^_</span>) of decoder layers </p>\n": "<p>Mapping (<span translate=no>_^_0_^_</span>) of decoder layers </p>\n",
|
||||
"<p>Move <span translate=no>_^_0_^_</span> to device </p>\n": "<p>Move <span translate=no>_^_0_^_</span> to device </p>\n",
|
||||
"<p>Move the parameters based on mapping </p>\n": "<p>Move the parameters based on mapping </p>\n",
|
||||
"<p>Optimize </p>\n": "<p>Optimize </p>\n",
|
||||
"<p>Optimizer </p>\n": "<p>Optimizer </p>\n",
|
||||
"<p>Training configs </p>\n": "<p>Training configs </p>\n",
|
||||
"<p>Transformer embedding and prediction layer parameter mapping (<span translate=no>_^_0_^_</span>) </p>\n": "<p>Transformer embedding and prediction layer parameter mapping (<span translate=no>_^_0_^_</span>) </p>\n",
|
||||
"<p>make sure that only lora weights are not loaded </p>\n": "<p>make sure that only lora weights are not loaded </p>\n",
|
||||
"Finetune GPT-2 with LoRA": "Finetune GPT-2 with LoRA",
|
||||
"This is training code with notes for fine-tuning pre-trained GPT-2 model with LoRA.": "This is training code with notes for fine-tuning pre-trained GPT-2 model with LoRA."
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"<h1>GPT-2 with <a href=\"index.html\">LoRA modules</a></h1>\n<p>Here's <a href=\"experiment.html\">the training code</a> for training a GPT2 model with LoRA on Tiny Shakespeare dataset.</p>\n": "<h1>GPT-2 with <a href=\"index.html\">LoRA modules</a></h1>\n<p>Here's <a href=\"experiment.html\">the training code</a> for training a GPT2 model with LoRA on Tiny Shakespeare dataset.</p>\n",
|
||||
"<h2>GPT2 Model</h2>\n": "<h2>GPT2 Model</h2>\n",
|
||||
"<h3>Decoder block</h3>\n": "<h3>Decoder block</h3>\n",
|
||||
"<h3>Feedforward Network</h3>\n": "<h3>Feedforward Network</h3>\n",
|
||||
"<h3>Multi-Head Attention</h3>\n": "<h3>Multi-Head Attention</h3>\n",
|
||||
"<p>Add position embeddings </p>\n": "<p>Add position embeddings </p>\n",
|
||||
"<p>Apply causal attention </p>\n": "<p>Apply causal attention </p>\n",
|
||||
"<p>Attention </p>\n": "<p>Attention </p>\n",
|
||||
"<p>Attention layer </p>\n": "<p>Attention layer </p>\n",
|
||||
"<p>Attention pre-normalization layer </p>\n": "<p>Attention pre-normalization layer </p>\n",
|
||||
"<p>Decoder blocks </p>\n": "<p>Decoder blocks </p>\n",
|
||||
"<p>FFN </p>\n": "<p>FFN </p>\n",
|
||||
"<p>FFN pre-normalization layer </p>\n": "<p>FFN pre-normalization layer </p>\n",
|
||||
"<p>Feed-forward network </p>\n": "<p>Feed-forward network </p>\n",
|
||||
"<p>Final layer norm </p>\n": "<p>Final layer norm </p>\n",
|
||||
"<p>Final normalization </p>\n": "<p>Final normalization </p>\n",
|
||||
"<p>Final project </p>\n": "<p>Final project </p>\n",
|
||||
"<p>Get logits from projection layer </p>\n": "<p>Get logits from projection layer </p>\n",
|
||||
"<p>Get position embeddings </p>\n": "<p>Get position embeddings </p>\n",
|
||||
"<p>Get position ids </p>\n": "<p>Get position ids </p>\n",
|
||||
"<p>Get query, key and value </p>\n": "<p>Get query, key and value </p>\n",
|
||||
"<p>Get token embeddings </p>\n": "<p>Get token embeddings </p>\n",
|
||||
"<p>Linear transformation for QKV </p>\n": "<p>Linear transformation for QKV </p>\n",
|
||||
"<p>Output projection </p>\n": "<p>Output projection </p>\n",
|
||||
"<p>Projection layer to logit space </p>\n": "<p>Projection layer to logit space </p>\n",
|
||||
"<p>Reorder to <span translate=no>_^_0_^_</span> </p>\n": "<p>Reorder to <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Run through transformer blocks </p>\n": "<p>Run through transformer blocks </p>\n",
|
||||
"<p>Split last dimension to <span translate=no>_^_0_^_</span> </p>\n": "<p>Split last dimension to <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>The linear layers and the activation </p>\n": "<p>The linear layers and the activation </p>\n",
|
||||
"<p>Token and absolute positional embeddings </p>\n": "<p>Token and absolute positional embeddings </p>\n",
|
||||
"<p>Transform them from shape <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>Transform them from shape <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the embeddings tensor with shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> is the embeddings tensor with shape <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of dimensions </li>\n<li><span translate=no>_^_1_^_</span> is the size of the hidden dimension </li>\n<li><span translate=no>_^_2_^_</span> is the lora rank</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> is the number of dimensions </li>\n<li><span translate=no>_^_1_^_</span> is the size of the hidden dimension </li>\n<li><span translate=no>_^_2_^_</span> is the lora rank</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of dimensions in the embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of decoder layers </li>\n<li><span translate=no>_^_3_^_</span> is the number of positional embeddings </li>\n<li><span translate=no>_^_4_^_</span> is the layer norm epsilon </li>\n<li><span translate=no>_^_5_^_</span> is the vocabulary size </li>\n<li><span translate=no>_^_6_^_</span> is the lora rank</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> is the number of dimensions in the embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of decoder layers </li>\n<li><span translate=no>_^_3_^_</span> is the number of positional embeddings </li>\n<li><span translate=no>_^_4_^_</span> is the layer norm epsilon </li>\n<li><span translate=no>_^_5_^_</span> is the vocabulary size </li>\n<li><span translate=no>_^_6_^_</span> is the lora rank</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of dimensions in the embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of heads </li>\n<li><span translate=no>_^_2_^_</span> is the layer norm epsilon </li>\n<li><span translate=no>_^_3_^_</span> is the lora rank</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> is the number of dimensions in the embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of heads </li>\n<li><span translate=no>_^_2_^_</span> is the layer norm epsilon </li>\n<li><span translate=no>_^_3_^_</span> is the lora rank</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of dimensions in the embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of heads </li>\n<li><span translate=no>_^_2_^_</span> is the lora rank</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> is the number of dimensions in the embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of heads </li>\n<li><span translate=no>_^_2_^_</span> is the lora rank</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the tensor with shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> is the tensor with shape <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"GPT-2 implementation with LoRA modules": "GPT-2 implementation with LoRA modules",
|
||||
"GPT-2 with LoRA": "GPT-2 with LoRA"
|
||||
}
|
||||
Reference in New Issue
Block a user