Here's the training code for training a GPT2 model with LoRA on Tiny Shakespeare dataset.
\n": "Here's the training code for training a GPT2 model with LoRA on Tiny Shakespeare dataset.
\n", "Add position embeddings
\n": "Add position embeddings
\n", "Apply causal attention
\n": "Apply causal attention
\n", "Attention
\n": "Attention
\n", "Attention layer
\n": "Attention layer
\n", "Attention pre-normalization layer
\n": "Attention pre-normalization layer
\n", "Decoder blocks
\n": "Decoder blocks
\n", "FFN
\n": "FFN
\n", "FFN pre-normalization layer
\n": "FFN pre-normalization layer
\n", "Feed-forward network
\n": "Feed-forward network
\n", "Final layer norm
\n": "Final layer norm
\n", "Final normalization
\n": "Final normalization
\n", "Final project
\n": "Final project
\n", "Get logits from projection layer
\n": "Get logits from projection layer
\n", "Get position embeddings
\n": "Get position embeddings
\n", "Get position ids
\n": "Get position ids
\n", "Get query, key and value
\n": "Get query, key and value
\n", "Get token embeddings
\n": "Get token embeddings
\n", "Linear transformation for QKV
\n": "Linear transformation for QKV
\n", "Output projection
\n": "Output projection
\n", "Projection layer to logit space
\n": "Projection layer to logit space
\n", "Reorder to _^_0_^_
\n": "Reorder to _^_0_^_
\n", "Run through transformer blocks
\n": "Run through transformer blocks
\n", "Split last dimension to _^_0_^_
\n": "Split last dimension to _^_0_^_
\n", "The linear layers and the activation
\n": "The linear layers and the activation
\n", "Token and absolute positional embeddings
\n": "Token and absolute positional embeddings
\n", "Transform them from shape _^_0_^_ to _^_1_^_
\n": "Transform them from shape _^_0_^_ to _^_1_^_
\n", "