wehub/labmlai--annotated_deep_learning_paper_implementations
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:19:01 +08:00
commit 3b90d1192f
2172 changed files with 594509 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
translate_cache/transformers/gpt/__init__.ja.json
+59
View File
@@ -0,0 +1,59 @@
{
"<h1>GPT</h1>\n<p>This is a tutorial/implementation of <a href=\"https://openai.com/blog/better-language-models/\">OpenAI GPT architecture</a> in <a href=\"https://pytorch.org\">PyTorch</a>. We got a bunch of implementation details from <a href=\"https://github.com/karpathy/minGPT\">minGPT</a> by <a href=\"https://twitter.com/karpathy\">@karpathy</a>. This implementation also uses character tiny shakespeare dataset.</p>\n<p>GPT model is essentially a standard transformer with a few tweaks. GPT-2 and especially GPT-3 models are quite large and won&#x27;t fit on a single GPU and will need model parallelism. This implementation doesn&#x27;t even use data parallelism and is intended to be more of a tutorial.</p>\n<p>Main differences of this compared to a simple autoregressive transformer are the parameter initialization, weight decay, and learning rate schedule. For the transformer we reuse the <a href=\"../transformers/index.html\">existing labml/nn transformer implementation</a>.</p>\n<p>Here&#x27;s a notebook for training a GPT model on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/gpt/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>GPT</h1>\n</a><p><a href=\"https://pytorch.org\">\u3053\u308c\u306f PyTorch \u306b\u304a\u3051\u308b <a href=\"https://openai.com/blog/better-language-models/\">OpenAI GPT \u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb/\u5b9f\u88c5\u3067\u3059\u3002</a><a href=\"https://twitter.com/karpathy\">@karpathy \u306b\u3088\u3063\u3066 <a href=\"https://github.com/karpathy/minGPT\">MingPT</a> \u304b\u3089\u5b9f\u88c5\u306e\u8a73\u7d30\u3092\u305f\u304f\u3055\u3093\u5f97\u307e\u3057\u305f\u3002</a>\u3053\u306e\u5b9f\u88c5\u3067\u306f\u3001\u6587\u5b57\u30b5\u30a4\u30ba\u306e\u5c0f\u3055\u3044\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3082\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</p>\u3002\n<p>GPT\u30e2\u30c7\u30eb\u306f\u57fa\u672c\u7684\u306b\u3001\u3044\u304f\u3064\u304b\u306e\u8abf\u6574\u3092\u52a0\u3048\u305f\u6a19\u6e96\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3067\u3059\u3002GPT-2\u3001\u7279\u306bGPT-3\u306e\u30e2\u30c7\u30eb\u306f\u975e\u5e38\u306b\u5927\u304d\u304f\u3001\u5358\u4e00\u306eGPU\u306b\u306f\u53ce\u307e\u3089\u306a\u3044\u305f\u3081\u3001\u30e2\u30c7\u30eb\u306e\u4e26\u5217\u51e6\u7406\u304c\u5fc5\u8981\u306b\u306a\u308a\u307e\u3059\u3002\u3053\u306e\u5b9f\u88c5\u306f\u30c7\u30fc\u30bf\u4e26\u5217\u51e6\u7406\u3059\u3089\u4f7f\u7528\u305b\u305a\u3001\u3069\u3061\u3089\u304b\u3068\u3044\u3046\u3068\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u306e\u3088\u3046\u306a\u3082\u306e\u3067\u3059</p>\u3002\n<p>\u5358\u7d14\u306a\u81ea\u5df1\u56de\u5e30\u5909\u63db\u5668\u3068\u306e\u4e3b\u306a\u9055\u3044\u306f\u3001\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u521d\u671f\u5316\u3001\u91cd\u307f\u306e\u6e1b\u8870\u3001\u5b66\u7fd2\u7387\u306e\u30b9\u30b1\u30b8\u30e5\u30fc\u30eb\u3067\u3059\u3002\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306b\u306f\u3001<a href=\"../transformers/index.html\">\u65e2\u5b58\u306elabml/nn\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u518d\u5229\u7528\u3057\u307e\u3059</a></p>\u3002\n<p>\u3053\u308c\u306f\u3001Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067GPT\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30ce\u30fc\u30c8\u30d6\u30c3\u30af\u3067\u3059\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/gpt/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u4ee5\u4e0b\u304b\u3089\u7d99\u627f\u3055\u308c\u307e\u3059 <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
"<h2>GPT model</h2>\n<p>This consists of a token embedding layer, transformer encoder, and a final linear layer that gives token logits.</p>\n": "<h2>GPT \u30e2\u30c7\u30eb</h2>\n<p>\u3053\u308c\u306f\u3001\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u5c64\u3001\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u3001\u304a\u3088\u3073\u30c8\u30fc\u30af\u30f3\u30ed\u30b8\u30c3\u30c8\u3092\u63d0\u4f9b\u3059\u308b\u6700\u5f8c\u306e\u7dda\u5f62\u5c64\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059\u3002</p>\n",
"<h3>Create custom optimizer with weight decay</h3>\n<p>This code is taken from <a href=\"https://github.com/karpathy/minGPT\">minGPT</a>. This applies weight decay only to weights of linear layers.</p>\n": "<h3>\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u3092\u542b\u3080\u30ab\u30b9\u30bf\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u4f5c\u6210</h3>\n<p><a href=\"https://github.com/karpathy/minGPT\">\u3053\u306e\u30b3\u30fc\u30c9\u306fMingPT\u304b\u3089\u53d6\u5f97\u3057\u305f\u3082\u306e\u3067\u3059</a>\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u306f\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc\u306e\u30a6\u30a7\u30a4\u30c8\u306b\u306e\u307f\u9069\u7528\u3055\u308c\u307e\u3059</p>\u3002\n",
"<h3>Initialize weights</h3>\n<p>Weights of linear layers and embedding layers are initialized to <span translate=no>_^_0_^_</span> instead of the default Xavier initialzation.</p>\n": "<h3>\u30a6\u30a7\u30a4\u30c8\u3092\u521d\u671f\u5316</h3>\n<p>\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc\u3068\u57cb\u3081\u8fbc\u307f\u30ec\u30a4\u30e4\u30fc\u306e\u91cd\u307f\u306f\u3001\u30c7\u30d5\u30a9\u30eb\u30c8\u306e Xavier <span translate=no>_^_0_^_</span> \u521d\u671f\u5316\u306e\u4ee3\u308f\u308a\u306b\u521d\u671f\u5316\u3055\u308c\u307e\u3059\u3002</p>\n",
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
"<p> </p>\n": "<p></p>\n",
"<p> Create GPT model and initialize weights</p>\n": "<p>GPT \u30e2\u30c7\u30eb\u306e\u4f5c\u6210\u3068\u91cd\u307f\u306e\u521d\u671f\u5316</p>\n",
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
"<p>Apply custom weight initialization </p>\n": "<p>\u30ab\u30b9\u30bf\u30e0\u30a6\u30a7\u30a4\u30c8\u521d\u671f\u5316\u3092\u9069\u7528</p>\n",
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
"<p>Collect names of parameters to apply weight decay </p>\n": "<p>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u540d\u524d\u3092\u53ce\u96c6\u3057\u3066\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u3092\u9069\u7528\u3059\u308b</p>\n",
"<p>Create a <a href=\"../optimizers/configs.html#OptimizerConfigs\">configurable optimizer</a>, so that we can change these simply by passing a config dictionary. </p>\n": "<p><a href=\"../optimizers/configs.html#OptimizerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f5c\u6210\u3057\u3066</a>\u3001\u8a2d\u5b9a\u8f9e\u66f8\u3092\u6e21\u3059\u3060\u3051\u3067\u3053\u308c\u3089\u3092\u5909\u66f4\u3067\u304d\u308b\u3088\u3046\u306b\u3057\u307e\u3059\u3002</p>\n",
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u30de\u30b9\u30af\u304c\u521d\u671f\u5316\u3055\u308c\u3066\u3044\u306a\u3044\u5834\u5408\u3084\u30de\u30b9\u30af\u306e\u30b5\u30a4\u30ba\u304c\u7570\u306a\u308b\u5834\u5408\u306f\u3001\u5f8c\u7d9a\u306e\u30de\u30b9\u30af\u3092\u4f5c\u6210\u3057\u307e\u3059</p>\n",
"<p>Custom optimizer </p>\n": "<p>\u30ab\u30b9\u30bf\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</p>\n",
"<p>GPT model </p>\n": "<p>GPT \u30e2\u30c7\u30eb</p>\n",
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT \u306f GELU \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u3092\u4f7f\u7528\u3057\u3066\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u3092\u884c\u3044\u307e\u3059</p>\n",
"<p>GPT uses a maximum learning rate of <span translate=no>_^_0_^_</span>. </p>\n": "<p>GPT \u306e\u6700\u5927\u5b66\u7fd2\u7387\u306f\u3067\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
"<p>Get all the parameters </p>\n": "<p>\u3059\u3079\u3066\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u53d6\u5f97</p>\n",
"<p>Get logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
"<p>Initialize biases to <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30a4\u30a2\u30b9\u3092\u521d\u671f\u5316 <span translate=no>_^_0_^_</span></p>\n",
"<p>Number of tokens for wamup </p>\n": "<p>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u7528\u306e\u30c8\u30fc\u30af\u30f3\u6570</p>\n",
"<p>Number of warmup optimization steps </p>\n": "<p>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6700\u9069\u5316\u30b9\u30c6\u30c3\u30d7\u306e\u6570</p>\n",
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
"<p>Parameters that are not decayed </p>\n": "<p>\u6e1b\u8870\u3057\u306a\u3044\u30d1\u30e9\u30e1\u30fc\u30bf</p>\n",
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3057\u307e\u3059\uff08\u30c8\u30ec\u30fc\u30ca\u30fc\u306fRNN\u3067\u3082\u4f7f\u7528\u3055\u308c\u308b\u305f\u3081\u30012\u756a\u76ee\u306e\u5024\u306f\u72b6\u614b\u7528\u3067\u3059\uff09</p>\n",
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
"<p>Set default weight decay. This is not required since we set the weight decay in the parameter groups. </p>\n": "<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u3067\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u3092\u8a2d\u5b9a\u3057\u3066\u3044\u308b\u306e\u3067\u3001\u3053\u308c\u306f\u5fc5\u9808\u3067\u306f\u3042\u308a\u307e\u305b\u3093</p>\u3002\n",
"<p>Set model embedding size, required if we use <a href=\"../optimizers/noam.html\">Noam optimizer</a> which has an exponential decay. </p>\n": "<p>\u30e2\u30c7\u30eb\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002\u6307\u6570\u95a2\u6570\u7684\u306b\u6e1b\u8870\u3059\u308b <a href=\"../optimizers/noam.html\">Noam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f7f\u7528\u3059\u308b\u5834\u5408\u306b\u5fc5\u8981\u3067\u3059</a>\u3002</p>\n",
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
"<p>Set parameter groups for optimization. </p>\n": "<p>\u6700\u9069\u5316\u7528\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002</p>\n",
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u6b21\u306b\u30de\u30b9\u30af\u3059\u308b\u3068\u3001\u30c8\u30fc\u30af\u30f3\u304c\u30de\u30b9\u30af\u3055\u308c\u3001\u5c06\u6765\u306e\u30c8\u30fc\u30af\u30f3\u304c\u898b\u3048\u306a\u304f\u306a\u308a\u307e\u3059</p>\n",
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
"<p>The mask will be initialized on the first call </p>\n": "<p>\u30de\u30b9\u30af\u306f\u6700\u521d\u306e\u547c\u3073\u51fa\u3057\u3067\u521d\u671f\u5316\u3055\u308c\u307e\u3059</p>\n",
"<p>Total number of optimization steps for learning rate cosine decay </p>\n": "<p>\u5b66\u7fd2\u7387\u30b3\u30b5\u30a4\u30f3\u6e1b\u8870\u306e\u6700\u9069\u5316\u30b9\u30c6\u30c3\u30d7\u306e\u7dcf\u6570</p>\n",
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
"<p>Transformer configurations </p>\n": "<p>\u5909\u5727\u5668\u69cb\u6210</p>\n",
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
"<p>Use <a href=\"../optimizers/adam_warmup_cosine_decay.html\">cosine decay optimizer</a>. This is what GPT uses. </p>\n": "<p><a href=\"../optimizers/adam_warmup_cosine_decay.html\">\u30b3\u30b5\u30a4\u30f3\u6e1b\u8870\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f7f\u7528\u3057\u3066\u304f\u3060\u3055\u3044</a>\u3002\u3053\u308c\u304cGPT\u304c\u4f7f\u7528\u3059\u308b\u3082\u306e\u3067\u3059</p>\u3002\n",
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
"<p>Weight decay </p>\n": "<p>\u4f53\u91cd\u6e1b\u5c11</p>\n",
"<p>Weight decay is decoupled from gradients </p>\n": "<p>\u91cd\u91cf\u306e\u6e1b\u8870\u306f\u52fe\u914d\u304b\u3089\u5207\u308a\u96e2\u3055\u308c\u307e\u3059</p>\n",
"<p>create the pytorch optimizer object </p>\n": "<p>pytorch \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u4f5c\u6210\u3059\u308b</p>\n",
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../models.html#Encoder\">\u5909\u5727\u5668\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span><a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u306f\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3059 (\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u4ed8\u304d)</a></li>\n</ul><li><span translate=no>_^_2_^_</span><a href=\"../models.html#Generator\">\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u6700\u5f8c\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u3067\u3059</a>\u3002</li>\n",
"GPT": "GPT",
"Implementation/tutorial of GPT model and training code.": "GPT\u30e2\u30c7\u30eb\u3068\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306e\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002"
}
translate_cache/transformers/gpt/__init__.si.json
+59
View File
File diff suppressed because one or more lines are too long
translate_cache/transformers/gpt/__init__.zh.json
+59
View File
@@ -0,0 +1,59 @@
{
"<h1>GPT</h1>\n<p>This is a tutorial/implementation of <a href=\"https://openai.com/blog/better-language-models/\">OpenAI GPT architecture</a> in <a href=\"https://pytorch.org\">PyTorch</a>. We got a bunch of implementation details from <a href=\"https://github.com/karpathy/minGPT\">minGPT</a> by <a href=\"https://twitter.com/karpathy\">@karpathy</a>. This implementation also uses character tiny shakespeare dataset.</p>\n<p>GPT model is essentially a standard transformer with a few tweaks. GPT-2 and especially GPT-3 models are quite large and won&#x27;t fit on a single GPU and will need model parallelism. This implementation doesn&#x27;t even use data parallelism and is intended to be more of a tutorial.</p>\n<p>Main differences of this compared to a simple autoregressive transformer are the parameter initialization, weight decay, and learning rate schedule. For the transformer we reuse the <a href=\"../transformers/index.html\">existing labml/nn transformer implementation</a>.</p>\n<p>Here&#x27;s a notebook for training a GPT model on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/gpt/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>GPT</h1>\n</a><p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u4e2d <a href=\"https://openai.com/blog/better-language-models/\">OpenAI GPT \u67b6\u6784\u7684\u6559\u7a0b/\u5b9e\u73b0\u3002<a href=\"https://twitter.com/karpathy\">@karpathy</a> \u4ece <a href=\"https://github.com/karpathy/minGPT\">MinGpt</a> \u90a3\u91cc\u5f97\u5230\u4e86\u5f88\u591a\u5b9e\u73b0\u7ec6\u8282\u3002\u6b64\u5b9e\u73b0\u8fd8\u4f7f\u7528\u89d2\u8272\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u3002</p>\n<p>GPT \u6a21\u578b\u672c\u8d28\u4e0a\u662f\u4e00\u4e2a\u6807\u51c6\u7684\u53d8\u538b\u5668\uff0c\u4f46\u6709\u4e00\u4e9b\u8c03\u6574\u3002GPT-2\uff0c\u5c24\u5176\u662f GPT-3 \u6a21\u578b\u975e\u5e38\u5927\uff0c\u4e0d\u9002\u5408\u5355\u4e2a GPU\uff0c\u9700\u8981\u6a21\u578b\u5e76\u884c\u5904\u7406\u3002\u6b64\u5b9e\u73b0\u751a\u81f3\u4e0d\u4f7f\u7528\u6570\u636e\u5e76\u884c\u6027\uff0c\u65e8\u5728\u66f4\u50cf\u662f\u4e00\u4e2a\u6559\u7a0b\u3002</p>\n\u4e0e@@ <p>\u7b80\u5355\u7684\u81ea\u56de\u5f52\u8f6c\u6362\u5668\u76f8\u6bd4\uff0c\u5176\u4e3b\u8981\u533a\u522b\u5728\u4e8e\u53c2\u6570\u521d\u59cb\u5316\u3001\u6743\u91cd\u8870\u51cf\u548c\u5b66\u4e60\u901f\u7387\u8c03\u5ea6\u3002\u5bf9\u4e8e\u53d8\u538b\u5668\uff0c\u6211\u4eec\u91cd\u7528\u4e86<a href=\"../transformers/index.html\">\u73b0\u6709\u7684 labml/nn \u53d8\u6362\u5668\u5b9e\u73b0</a>\u3002</p>\n<p>\u8fd9\u662f\u4e00\u672c\u7528\u4e8e\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3 GPT \u6a21\u578b\u7684\u7b14\u8bb0\u672c\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/gpt/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
"<h2>GPT model</h2>\n<p>This consists of a token embedding layer, transformer encoder, and a final linear layer that gives token logits.</p>\n": "<h2>GPT \u578b\u53f7</h2>\n<p>\u8fd9\u5305\u62ec\u4ee4\u724c\u5d4c\u5165\u5c42\u3001\u53d8\u538b\u5668\u7f16\u7801\u5668\u548c\u7ed9\u51fa\u4ee4\u724c\u65e5\u5fd7\u7684\u6700\u7ec8\u7ebf\u6027\u5c42\u3002</p>\n",
"<h3>Create custom optimizer with weight decay</h3>\n<p>This code is taken from <a href=\"https://github.com/karpathy/minGPT\">minGPT</a>. This applies weight decay only to weights of linear layers.</p>\n": "<h3>\u521b\u5efa\u5177\u6709\u6743\u91cd\u8870\u51cf\u7684\u81ea\u5b9a\u4e49\u4f18\u5316\u5668</h3>\n<p>\u6b64\u4ee3\u7801\u53d6\u81ea <a href=\"https://github.com/karpathy/minGPT\">MingPT</a>\u3002\u8fd9\u4ec5\u5c06\u6743\u91cd\u8870\u51cf\u5e94\u7528\u4e8e\u7ebf\u6027\u56fe\u5c42\u7684\u6743\u91cd\u3002</p>\n",
"<h3>Initialize weights</h3>\n<p>Weights of linear layers and embedding layers are initialized to <span translate=no>_^_0_^_</span> instead of the default Xavier initialzation.</p>\n": "<h3>\u521d\u59cb\u5316\u6743\u91cd</h3>\n<p>\u7ebf\u6027\u5c42\u548c\u5d4c\u5165\u5c42\u7684\u6743\u91cd\u521d\u59cb\u5316\u4e3a\uff0c<span translate=no>_^_0_^_</span>\u800c\u4e0d\u662f\u9ed8\u8ba4\u7684 Xavier \u521d\u59cb\u5316\u3002</p>\n",
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
"<p> </p>\n": "<p></p>\n",
"<p> Create GPT model and initialize weights</p>\n": "<p>\u521b\u5efa GPT \u6a21\u578b\u5e76\u521d\u59cb\u5316\u6743\u91cd</p>\n",
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
"<p>Apply custom weight initialization </p>\n": "<p>\u5e94\u7528\u81ea\u5b9a\u4e49\u6743\u91cd\u521d\u59cb\u5316</p>\n",
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
"<p>Collect names of parameters to apply weight decay </p>\n": "<p>\u6536\u96c6\u53c2\u6570\u540d\u79f0\u4ee5\u5e94\u7528\u6743\u91cd\u8870\u51cf</p>\n",
"<p>Create a <a href=\"../optimizers/configs.html#OptimizerConfigs\">configurable optimizer</a>, so that we can change these simply by passing a config dictionary. </p>\n": "<p>\u521b\u5efa\u4e00\u4e2a<a href=\"../optimizers/configs.html#OptimizerConfigs\">\u53ef\u914d\u7f6e\u7684\u4f18\u5316\u5668</a>\uff0c\u8fd9\u6837\u6211\u4eec\u5c31\u53ef\u4ee5\u901a\u8fc7\u4f20\u9012\u914d\u7f6e\u5b57\u5178\u6765\u66f4\u6539\u5b83\u4eec\u3002</p>\n",
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u5982\u679c\u63a9\u7801\u672a\u521d\u59cb\u5316\u6216\u63a9\u7801\u5927\u5c0f\u4e0d\u540c\uff0c\u5219\u521b\u5efa\u540e\u7eed\u63a9\u7801</p>\n",
"<p>Custom optimizer </p>\n": "<p>\u81ea\u5b9a\u4e49\u4f18\u5316\u5668</p>\n",
"<p>GPT model </p>\n": "<p>GPT \u578b\u53f7</p>\n",
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT \u4f7f\u7528 GELU \u6fc0\u6d3b\u8fdb\u884c\u4f4d\u7f6e\u660e\u667a\u524d\u9988</p>\n",
"<p>GPT uses a maximum learning rate of <span translate=no>_^_0_^_</span>. </p>\n": "<p>GPT \u4f7f\u7528\u7684\u6700\u5927\u5b66\u4e60\u901f\u7387\u4e3a<span translate=no>_^_0_^_</span>\u3002</p>\n",
"<p>Get all the parameters </p>\n": "<p>\u83b7\u53d6\u6240\u6709\u53c2\u6570</p>\n",
"<p>Get logits </p>\n": "<p>\u83b7\u53d6\u65e5\u5fd7</p>\n",
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f7f\u7528\u4f4d\u7f6e\u7f16\u7801\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
"<p>Initialize biases to <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5c06\u504f\u5dee\u521d\u59cb\u5316\u4e3a<span translate=no>_^_0_^_</span></p>\n",
"<p>Number of tokens for wamup </p>\n": "<p>wamup \u7684\u4ee3\u5e01\u6570\u91cf</p>\n",
"<p>Number of warmup optimization steps </p>\n": "<p>\u9884\u70ed\u4f18\u5316\u6b65\u9aa4\u6570</p>\n",
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
"<p>Parameters that are not decayed </p>\n": "<p>\u672a\u8870\u51cf\u7684\u53c2\u6570</p>\n",
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c\uff08\u7b2c\u4e8c\u4e2a\u503c\u7528\u4e8e\u72b6\u6001\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u8bad\u7ec3\u5668\u4e5f\u4e0e RNN \u4e00\u8d77\u4f7f\u7528\uff09</p>\n",
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
"<p>Set default weight decay. This is not required since we set the weight decay in the parameter groups. </p>\n": "<p>\u8bbe\u7f6e\u9ed8\u8ba4\u6743\u91cd\u8870\u51cf\u3002\u8fd9\u4e0d\u662f\u5fc5\u9700\u7684\uff0c\u56e0\u4e3a\u6211\u4eec\u5728\u53c2\u6570\u7ec4\u4e2d\u8bbe\u7f6e\u4e86\u6743\u91cd\u8870\u51cf\u3002</p>\n",
"<p>Set model embedding size, required if we use <a href=\"../optimizers/noam.html\">Noam optimizer</a> which has an exponential decay. </p>\n": "<p>\u8bbe\u7f6e\u6a21\u578b\u5d4c\u5165\u5927\u5c0f\uff0c\u5982\u679c\u6211\u4eec\u4f7f\u7528\u5177\u6709\u6307\u6570\u8870\u51cf\u7684 <a href=\"../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a>\uff0c\u5219\u9700\u8981\u8bbe\u7f6e\u6a21\u578b\u5d4c\u5165\u5927\u5c0f\u3002</p>\n",
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
"<p>Set parameter groups for optimization. </p>\n": "<p>\u8bbe\u7f6e\u53c2\u6570\u7ec4\u4ee5\u8fdb\u884c\u4f18\u5316\u3002</p>\n",
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u540e\u7eed\u7684\u63a9\u7801\uff0c\u5c06\u63a9\u76d6\u4ee4\u724c\u4ee5\u514d\u770b\u5230\u672a\u6765\u7684\u4ee3\u5e01</p>\n",
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
"<p>The mask will be initialized on the first call </p>\n": "<p>\u63a9\u7801\u5c06\u5728\u7b2c\u4e00\u6b21\u8c03\u7528\u65f6\u521d\u59cb\u5316</p>\n",
"<p>Total number of optimization steps for learning rate cosine decay </p>\n": "<p>\u5b66\u4e60\u901f\u7387\u4f59\u5f26\u8870\u51cf\u7684\u4f18\u5316\u6b65\u9aa4\u603b\u6570</p>\n",
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
"<p>Transformer configurations </p>\n": "<p>\u53d8\u538b\u5668\u914d\u7f6e</p>\n",
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
"<p>Use <a href=\"../optimizers/adam_warmup_cosine_decay.html\">cosine decay optimizer</a>. This is what GPT uses. </p>\n": "<p>\u4f7f\u7528<a href=\"../optimizers/adam_warmup_cosine_decay.html\">\u4f59\u5f26\u8870\u51cf\u4f18\u5316\u5668</a>\u3002\u8fd9\u5c31\u662f GPT \u4f7f\u7528\u7684\u3002</p>\n",
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
"<p>Weight decay </p>\n": "<p>\u4f53\u91cd\u8870\u51cf</p>\n",
"<p>Weight decay is decoupled from gradients </p>\n": "<p>\u6743\u91cd\u8870\u51cf\u4e0e\u68af\u5ea6\u5206\u79bb</p>\n",
"<p>create the pytorch optimizer object </p>\n": "<p>\u521b\u5efa pytorch \u4f18\u5316\u5668\u5bf9\u8c61</p>\n",
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668<a href=\"../models.html#Encoder\">\u7f16\u7801\u5668</a></li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c<a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u5d4c\u5165\u6a21\u5757\uff08\u5e26\u6709\u4f4d\u7f6e\u7f16\u7801\uff09</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u7ed9<a href=\"../models.html#Generator\">\u51fa logit \u7684\u6700\u540e\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42</a>\u3002</li></ul>\n",
"GPT": "GPT",
"Implementation/tutorial of GPT model and training code.": "GPT \u6a21\u578b\u548c\u8bad\u7ec3\u4ee3\u7801\u7684\u5b9e\u73b0/\u6559\u7a0b\u3002"
}