This inherits from _^_0_^_
\n": "This inherits from _^_0_^_
\n", "\n": "
\n", "
Create RWKV model and initialize weights
\n": "Create RWKV model and initialize weights
\n", "Apply custom weight initialization
\n": "Apply custom weight initialization
\n", "Batch size _^_0_^_
\n": "Batch size _^_0_^_
\n", "Create AdamW optimizer and use the fused version if it is available
\n": "Create AdamW optimizer and use the fused version if it is available
\n", "Create configs
\n": "Create configs
\n", "Create experiment
\n": "Create experiment
\n", "Custom optimizer
\n": "Custom optimizer
\n", "Override configurations
\n": "Override configurations
\n", "Prompt separator is blank
\n": "Prompt separator is blank
\n", "RWKV model
\n": "RWKV model
\n", "Run training
\n": "Run training
\n", "Set models for saving and loading
\n": "Set models for saving and loading
\n", "Set the vocabulary sizes for embeddings and generating logits
\n": "Set the vocabulary sizes for embeddings and generating logits
\n", "Start the experiment
\n": "Start the experiment
\n", "Starting prompt for sampling
\n": "Starting prompt for sampling
\n", "Switch between training and validation for _^_0_^_ times per epoch
\n": "Switch between training and validation for _^_0_^_ times per epoch
\n", "Train for _^_0_^_ epochs
\n": "Train for _^_0_^_ epochs
\n", "Use Tiny Shakespeare dataset
\n": "Use Tiny Shakespeare dataset
\n", "Use a context size of _^_0_^_
\n": "Use a context size of _^_0_^_
\n", "Use character level tokenizer
\n": "Use character level tokenizer
\n", "We use our configurable RWKV implementation
\n": "We use our configurable RWKV implementation
\n", "create optim groups. Any parameters that is 2D will be weight decayed, otherwise no. i.e. all weight tensors in matmuls + embeddings decay, all biases and layernorms don't.
\n": "create optim groups. Any parameters that is 2D will be weight decayed, otherwise no. i.e. all weight tensors in matmuls + embeddings decay, all biases and layernorms don't.
\n", "filter out those that do not require grad
\n": "filter out those that do not require grad
\n", "initialize Vector Parameters in TimeMixing
\n": "initialize Vector Parameters in TimeMixing
\n", "model
\n": "model
\n", "number of warmup iterations
\n": "number of warmup iterations
\n", "start with all of the candidate parameters
\n": "start with all of the candidate parameters
\n", "total number of training iterations
\n": "total number of training iterations
\n", "weight decay
\n": "weight decay
\n", "experiment.py": "experiment.py" }