chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1>Transformer Auto-Regression Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset).</p>\n": "<h1>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u52d5\u56de\u5e30\u5b9f\u9a13</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/1706.03762\">\u5fc5\u8981\u306a\u306e\u306f\u6ce8\u610f\u3060\u3051</a>\u300d\u3067\u7d39\u4ecb\u3057\u305f\u30b7\u30f3\u30d7\u30eb\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092NLP\u81ea\u52d5\u56de\u5e30\u30bf\u30b9\u30af\uff08Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u7528\uff09\u3067\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n": "<h2>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u4ee5\u4e0b\u304b\u3089\u7d99\u627f\u3055\u308c\u307e\u3059 <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create GPT model and initialize weights</p>\n": "<p>GPT \u30e2\u30c7\u30eb\u306e\u4f5c\u6210\u3068\u91cd\u307f\u306e\u521d\u671f\u5316</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u30de\u30b9\u30af\u304c\u521d\u671f\u5316\u3055\u308c\u3066\u3044\u306a\u3044\u5834\u5408\u3084\u30de\u30b9\u30af\u306e\u30b5\u30a4\u30ba\u304c\u7570\u306a\u308b\u5834\u5408\u306f\u3001\u5f8c\u7d9a\u306e\u30de\u30b9\u30af\u3092\u4f5c\u6210\u3057\u307e\u3059</p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT \u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3057\u307e\u3059\uff08\u30c8\u30ec\u30fc\u30ca\u30fc\u306fRNN\u3067\u3082\u4f7f\u7528\u3055\u308c\u308b\u305f\u3081\u30012\u756a\u76ee\u306e\u5024\u306f\u72b6\u614b\u7528\u3067\u3059\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u6b21\u306b\u30de\u30b9\u30af\u3059\u308b\u3068\u3001\u30c8\u30fc\u30af\u30f3\u304c\u30de\u30b9\u30af\u3055\u308c\u3001\u5c06\u6765\u306e\u30c8\u30fc\u30af\u30f3\u304c\u898b\u3048\u306a\u304f\u306a\u308a\u307e\u3059</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u30de\u30b9\u30af\u306f\u6700\u521d\u306e\u547c\u3073\u51fa\u3057\u3067\u521d\u671f\u5316\u3055\u308c\u307e\u3059</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32 \u30a8\u30dd\u30c3\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../models.html#Encoder\">\u5909\u5727\u5668\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span><a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u306f\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3059 (\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u4ed8\u304d)</a></li>\n</ul><li><span translate=no>_^_2_^_</span><a href=\"../models.html#Generator\">\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u6700\u5f8c\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u3067\u3059</a>\u3002</li>\n",
|
||||
"This trains a simple transformer model on NLP auto-regression.": "\u3053\u308c\u306b\u3088\u308a\u3001\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u306b NLP \u81ea\u52d5\u56de\u5e30\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u884c\u3044\u307e\u3059\u3002",
|
||||
"Transformer Auto-Regression Experiment": "\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u52d5\u56de\u5e30\u5b9f\u9a13"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1>Transformer Auto-Regression Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset).</p>\n": "<h1>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u0db8\u0dd9\u0dba \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd3 \u0d87\u0dad\u0dd2 \u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2 <a href=\"https://arxiv.org/abs/1706.03762\">\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba-\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0d94\u0db6\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc3\u0dd2\u0dba\u0dbd\u0dca\u0dbd</a> (\u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0dc3\u0db8\u0d9f).</p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0db1\u0dca\u0db1\u0dda <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create GPT model and initialize weights</p>\n": "<p> GPT\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 \u0dc3\u0dc4 \u0db6\u0dbb \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db1\u0db8\u0dca \u0dc4\u0ddd \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd9\u0dc4\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0db1\u0db8\u0dca \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dda\u0dad\u0db1 \u0d9a\u0dca\u0dbb\u0db8 \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0dcf\u0db7\u0db4\u0dca\u0dbb\u0dad\u0dd2 results \u0dbd (\u0daf\u0dd9\u0dc0\u0db1 \u0d85\u0d9c\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dda, \u0db8\u0db1\u0dca\u0daf \u0d85\u0db4\u0d9c\u0dda \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4 RNs \u0dc3\u0db8\u0d9f \u0daf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2) </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab, \u0d85\u0db1\u0dcf\u0d9c\u0dad \u0da7\u0ddd\u0d9a\u0db1 \u0daf\u0dd0\u0d9a\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dc3\u0d82 \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0d87\u0db8\u0dad\u0dd4\u0db8\u0dd9\u0db1\u0dca \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad </p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32\u0dc0\u0dba\u0dc3 \u0d85\u0dc0\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"../models.html#Encoder\">\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a> </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0da7\u0ddd\u0d9a\u0db1\u0dca <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba (\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab \u0dc3\u0db8\u0d9f)</a> </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1 <a href=\"../models.html#Generator\">\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dba\u0dd2</a> . </li></ul>\n",
|
||||
"This trains a simple transformer model on NLP auto-regression.": "\u0db8\u0dd9\u0dba \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3\u0dad\u0dca\u0dc0\u0dba \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2.",
|
||||
"Transformer Auto-Regression Experiment": "\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1>Transformer Auto-Regression Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset).</p>\n": "<h1>\u53d8\u538b\u5668\u81ea\u52a8\u56de\u5f52\u5b9e\u9a8c</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u8fd9\u5c06\u8bad\u7ec3\u4e00\u4e2a\u5728 NLP \u81ea\u52a8\u56de\u5f52\u4efb\u52a1\uff08\u4f7f\u7528 Tiny Shakespeare \u6570\u636e\u96c6\uff09\u4e2d\u5f15\u5165\u7684 \u201c<a href=\"https://arxiv.org/abs/1706.03762\">\u6ce8\u610f\u529b\u5c31\u662f\u4f60\u6240\u9700\u8981</a>\u7684\u201d \u7b80\u5355\u53d8\u538b\u5668\u3002</p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n": "<h2>\u81ea\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create GPT model and initialize weights</p>\n": "<p>\u521b\u5efa GPT \u6a21\u578b\u5e76\u521d\u59cb\u5316\u6743\u91cd</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u5982\u679c\u63a9\u7801\u672a\u521d\u59cb\u5316\u6216\u63a9\u7801\u5927\u5c0f\u4e0d\u540c\uff0c\u5219\u521b\u5efa\u540e\u7eed\u63a9\u7801</p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT \u578b\u53f7</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u83b7\u53d6\u65e5\u5fd7</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f7f\u7528\u4f4d\u7f6e\u7f16\u7801\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u578b\u53f7\u5c3a\u5bf8</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c\uff08\u7b2c\u4e8c\u4e2a\u503c\u7528\u4e8e\u72b6\u6001\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u8bad\u7ec3\u5668\u4e5f\u4e0e RNN \u4e00\u8d77\u4f7f\u7528\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u540e\u7eed\u7684\u63a9\u7801\uff0c\u5c06\u63a9\u76d6\u4ee4\u724c\u4ee5\u514d\u770b\u5230\u672a\u6765\u7684\u4ee3\u5e01</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u63a9\u7801\u5c06\u5728\u7b2c\u4e00\u6b21\u8c03\u7528\u65f6\u521d\u59cb\u5316</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>\u8bad\u7ec3 32 \u4e2a\u65f6\u4ee3</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668<a href=\"../models.html#Encoder\">\u7f16\u7801\u5668</a></li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c<a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u5d4c\u5165\u6a21\u5757\uff08\u5e26\u6709\u4f4d\u7f6e\u7f16\u7801\uff09</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u7ed9<a href=\"../models.html#Generator\">\u51fa logit \u7684\u6700\u540e\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42</a>\u3002</li></ul>\n",
|
||||
"This trains a simple transformer model on NLP auto-regression.": "\u8fd9\u4f1a\u5728 NLP \u81ea\u52a8\u56de\u5f52\u4e0a\u8bad\u7ec3\u4e00\u4e2a\u7b80\u5355\u7684\u53d8\u538b\u5668\u6a21\u578b\u3002",
|
||||
"Transformer Auto-Regression Experiment": "\u53d8\u538b\u5668\u81ea\u52a8\u56de\u5f52\u5b9e\u9a8c"
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"<h1>Transformer Auto-Regression Experiment with <a href=\"../../optimizers/sophia.html\">Sophia-G optimizer</a></h1>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset) with <a href=\"../../optimizers/sophia.html\">Sophia-G optimizer</a>.</p>\n": "<h1>Transformer Auto-Regression Experiment with <a href=\"../../optimizers/sophia.html\">Sophia-G optimizer</a></h1>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset) with <a href=\"../../optimizers/sophia.html\">Sophia-G optimizer</a>.</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"autoregressive_experiment.html\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>Configurations</h2>\n<p>This inherits from <a href=\"autoregressive_experiment.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Training or validation step with Gauss-Newton-Bartlett (GNB) Hessian diagonal estimator</h3>\n": "<h3>Training or validation step with Gauss-Newton-Bartlett (GNB) Hessian diagonal estimator</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>Batch size <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>Calculate and log accuracy </p>\n",
|
||||
"<p>Calculate and log loss </p>\n": "<p>Calculate and log loss </p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>Calculate gradients </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>Clear the gradients </p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>Clip gradients </p>\n",
|
||||
"<p>Create a categorical distribution from logits </p>\n": "<p>Create a categorical distribution from logits </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>Create configs </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>Create experiment </p>\n",
|
||||
"<p>Estimate the Hessian diagonal every <span translate=no>_^_0_^_</span> steps </p>\n": "<p>Estimate the Hessian diagonal every <span translate=no>_^_0_^_</span> steps </p>\n",
|
||||
"<p>Get model outputs </p>\n": "<p>Get model outputs </p>\n",
|
||||
"<p>Get model outputs. It's returning a tuple for states when using RNNs. This is not implemented yet. \ud83d\ude1c </p>\n": "<p>Get model outputs. It's returning a tuple for states when using RNNs. This is not implemented yet. \ud83d\ude1c </p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>Log the model parameters and gradients on last batch of every epoch </p>\n",
|
||||
"<p>Model size </p>\n": "<p>Model size </p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>Move data to the device </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>Override configurations </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>Prompt separator is blank </p>\n",
|
||||
"<p>Run training </p>\n": "<p>Run training </p>\n",
|
||||
"<p>Sample <span translate=no>_^_0_^_</span> </p>\n": "<p>Sample <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>Save the tracked metrics </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>Set models for saving and loading </p>\n",
|
||||
"<p>Set training/eval mode </p>\n": "<p>Set training/eval mode </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>Start the experiment </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>Starting prompt for sampling </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>Take optimizer step </p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>Train for 32 epochs </p>\n",
|
||||
"<p>Train the model </p>\n": "<p>Train the model </p>\n",
|
||||
"<p>Update EMA Hessian diagonal</p>\n<span translate=no>_^_0_^_</span><p> </p>\n": "<p>Update EMA Hessian diagonal</p>\n<span translate=no>_^_0_^_</span><p> </p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>Update global step (number of tokens processed) when in training mode </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/sophia.html\">Sophia optimizer</a> </p>\n": "<p>Use <a href=\"../../optimizers/sophia.html\">Sophia optimizer</a> </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>Use Tiny Shakespeare dataset </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>Use character level tokenizer </p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>Whether to capture model outputs </p>\n",
|
||||
"This trains a simple transformer model on NLP auto-regression with Sophia-G optimizer.": "This trains a simple transformer model on NLP auto-regression with Sophia-G optimizer.",
|
||||
"Transformer Auto-Regression Experiment with [Sophia-G optimizer](../../optimizers/sophia.html)": "Transformer Auto-Regression Experiment with [Sophia-G optimizer](../../optimizers/sophia.html)"
|
||||
}
|
||||
Reference in New Issue
Block a user