chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"<h1>Receptance Weighted Key Value (RWKV)</h1>\n<p>This is a tutorial/implementation of RWKV from paper <a href=\"https://arxiv.org/pdf/2305.13048.pdf\">RWKV: Reinventing RNNs for the Transformer Era</a> in <a href=\"https://pytorch.org/\">PyTorch</a>.</p>\n<p>Full definition of a RWKV Language Model, all of it in this single file. References: 1) <a href=\"https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v4neo/src/model.py\">the official RWKV PyTorch implementation released by Bo Peng</a> 2) <a href=\"https://github.com/huggingface/transformers/blob/main/src/transformers/models/rwkv/modeling_rwkv.py\">huggingface/transformers PyTorch implementation</a></p>\n": "<h1>Receptance Weighted Key Value (RWKV)</h1>\n<p>This is a tutorial/implementation of RWKV from paper <a href=\"https://arxiv.org/pdf/2305.13048.pdf\">RWKV: Reinventing RNNs for the Transformer Era</a> in <a href=\"https://pytorch.org/\">PyTorch</a>.</p>\n<p>Full definition of a RWKV Language Model, all of it in this single file. References: 1) <a href=\"https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v4neo/src/model.py\">the official RWKV PyTorch implementation released by Bo Peng</a> 2) <a href=\"https://github.com/huggingface/transformers/blob/main/src/transformers/models/rwkv/modeling_rwkv.py\">huggingface/transformers PyTorch implementation</a></p>\n",
|
||||
"<h1>x = (Batch,Time,Channel)</h1>\n": "<h1>x = (Batch,Time,Channel)</h1>\n",
|
||||
"<h2>RWKV block element</h2>\n": "<h2>RWKV block element</h2>\n",
|
||||
"<h2>RWKV</h2>\n": "<h2>RWKV</h2>\n",
|
||||
"<h3>Channel Mixing</h3>\n": "<h3>Channel Mixing</h3>\n",
|
||||
"<h3>L2 loss wrapper</h3>\n<p><a href=\"https://github.com/BlinkDL/RWKV-LM/blob/cca1b5e8e597cf40675882bb10b46287c844e35c/RWKV-v4/src/model.py#L21\">ref</a></p>\n": "<h3>L2 loss wrapper</h3>\n<p><a href=\"https://github.com/BlinkDL/RWKV-LM/blob/cca1b5e8e597cf40675882bb10b46287c844e35c/RWKV-v4/src/model.py#L21\">ref</a></p>\n",
|
||||
"<h3>Layer normalization with bias</h3>\n": "<h3>Layer normalization with bias</h3>\n",
|
||||
"<h3>Time Mixing</h3>\n": "<h3>Time Mixing</h3>\n",
|
||||
"<p> x = (Batch,Time,Channel)</p>\n": "<p> x = (Batch,Time,Channel)</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Embedding Layer </p>\n": "<p>Embedding Layer </p>\n",
|
||||
"<p>Initiate model layers </p>\n": "<p>Initiate model layers </p>\n",
|
||||
"<p>Layer Norm </p>\n": "<p>Layer Norm </p>\n",
|
||||
"<p>Learnable Matrix </p>\n": "<p>Learnable Matrix </p>\n",
|
||||
"<p>Learnable Vector </p>\n": "<p>Learnable Vector </p>\n",
|
||||
"<p>Logit Layer and loss Function (for training) </p>\n": "<p>Logit Layer and loss Function (for training) </p>\n",
|
||||
"<p>Output linear layer </p>\n": "<p>Output linear layer </p>\n",
|
||||
"<p>RWKV Blocks </p>\n": "<p>RWKV Blocks </p>\n",
|
||||
"<p>Return Logits and loss </p>\n": "<p>Return Logits and loss </p>\n",
|
||||
"<p>Update state for next iteration </p>\n": "<p>Update state for next iteration </p>\n",
|
||||
"<p>WKV calculation </p>\n": "<p>WKV calculation </p>\n",
|
||||
"<p>channel mixing </p>\n": "<p>channel mixing </p>\n",
|
||||
"<p>if we are given some desired targets also calculate the loss </p>\n": "<p>if we are given some desired targets also calculate the loss </p>\n",
|
||||
"<p>inference-time mini-optimization: only forward the lm_head on the very last position </p>\n": "<p>inference-time mini-optimization: only forward the lm_head on the very last position </p>\n",
|
||||
"<p>learnable matrix </p>\n": "<p>learnable matrix </p>\n",
|
||||
"<p>learnable vector </p>\n": "<p>learnable vector </p>\n",
|
||||
"<p>state: <a href=\"batch_size, 5 , n_embd\">batch_size, 5 , n_embd</a> </p>\n": "<p>state: <a href=\"batch_size, 5 , n_embd\">batch_size, 5 , n_embd</a> </p>\n",
|
||||
"<p>time mixing </p>\n": "<p>time mixing </p>\n",
|
||||
"<p>to encourage the logits to be close to 0 </p>\n": "<p>to encourage the logits to be close to 0 </p>\n",
|
||||
"<p>token shifting </p>\n": "<p>token shifting </p>\n",
|
||||
"<p>update states </p>\n": "<p>update states </p>\n",
|
||||
"Receptance Weighted Key Value (RWKV)": "Receptance Weighted Key Value (RWKV)",
|
||||
"This implements the RWKV model using PyTorch with explanations.": "This implements the RWKV model using PyTorch with explanations."
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"<h2>Transformer Configurations</h2>\n<p>This defines configurations for a transformer. The configurations are calculate using option functions. These are lazy loaded and therefore only the necessary modules are calculated.</p>\n": "<h2>Transformer Configurations</h2>\n<p>This defines configurations for a transformer. The configurations are calculate using option functions. These are lazy loaded and therefore only the necessary modules are calculated.</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>Dropout probability </p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>Number of attention heads </p>\n",
|
||||
"<p>Number of layers </p>\n": "<p>Number of layers </p>\n",
|
||||
"<p>Number of tokens in the source vocabulary (for token embeddings) </p>\n": "<p>Number of tokens in the source vocabulary (for token embeddings) </p>\n",
|
||||
"<p>Number of tokens in the target vocabulary (to generate logits for prediction) </p>\n": "<p>Number of tokens in the target vocabulary (to generate logits for prediction) </p>\n",
|
||||
"<p>Transformer embedding size </p>\n": "<p>Transformer embedding size </p>\n",
|
||||
"configs.py": "configs.py"
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>RWKV configurations</h3>\n": "<h3>RWKV configurations</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create RWKV model and initialize weights</p>\n": "<p> Create RWKV model and initialize weights</p>\n",
|
||||
"<p>Apply custom weight initialization </p>\n": "<p>Apply custom weight initialization </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>Batch size <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create AdamW optimizer and use the fused version if it is available </p>\n": "<p>Create AdamW optimizer and use the fused version if it is available </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>Create configs </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>Create experiment </p>\n",
|
||||
"<p>Custom optimizer </p>\n": "<p>Custom optimizer </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>Override configurations </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>Prompt separator is blank </p>\n",
|
||||
"<p>RWKV model </p>\n": "<p>RWKV model </p>\n",
|
||||
"<p>Run training </p>\n": "<p>Run training </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>Set models for saving and loading </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>Set the vocabulary sizes for embeddings and generating logits </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>Start the experiment </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>Starting prompt for sampling </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>Use Tiny Shakespeare dataset </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>Use character level tokenizer </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#RWKVConfigs\">configurable RWKV implementation</a> </p>\n": "<p>We use our <a href=\"../configs.html#RWKVConfigs\">configurable RWKV implementation</a> </p>\n",
|
||||
"<p>create optim groups. Any parameters that is 2D will be weight decayed, otherwise no. i.e. all weight tensors in matmuls + embeddings decay, all biases and layernorms don't. </p>\n": "<p>create optim groups. Any parameters that is 2D will be weight decayed, otherwise no. i.e. all weight tensors in matmuls + embeddings decay, all biases and layernorms don't. </p>\n",
|
||||
"<p>filter out those that do not require grad </p>\n": "<p>filter out those that do not require grad </p>\n",
|
||||
"<p>initialize Vector Parameters in TimeMixing </p>\n": "<p>initialize Vector Parameters in TimeMixing </p>\n",
|
||||
"<p>model </p>\n": "<p>model </p>\n",
|
||||
"<p>number of warmup iterations </p>\n": "<p>number of warmup iterations </p>\n",
|
||||
"<p>start with all of the candidate parameters </p>\n": "<p>start with all of the candidate parameters </p>\n",
|
||||
"<p>total number of training iterations </p>\n": "<p>total number of training iterations </p>\n",
|
||||
"<p>weight decay </p>\n": "<p>weight decay </p>\n",
|
||||
"experiment.py": "experiment.py"
|
||||
}
|
||||
Reference in New Issue
Block a user