chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"<h1>Switch Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a switch transformer.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/switch/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u30b9\u30a4\u30c3\u30c1\u30c8\u30e9\u30f3\u30b9\u5b9f\u9a13</h1>\n<p>\u3053\u308c\u306f\u3001\u30b9\u30a4\u30c3\u30c1\u30c8\u30e9\u30f3\u30b9\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304dPyTorch\u5b9f\u9a13\u3067\u3059\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/switch/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52d5\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This extends <a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a>.</p>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u5e83\u304c\u308a\u307e\u3059\u3002<a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u8a2d\u5b9a\u306f\u3001\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u305f\u3068\u304d\u306b\u4e0a\u66f8\u304d\u3067\u304d\u3001\u307e\u305f\u4e0a\u66f8\u304d\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u3092\u521d\u671f\u5316</h3>\n",
|
||||
"<h3>Initialize the switch transformer</h3>\n": "<h3>\u30b9\u30a4\u30c3\u30c1\u30c8\u30e9\u30f3\u30b9\u3092\u521d\u671f\u5316\u3057\u307e\u3059</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u5b9f\u9a13\u3092\u5b9f\u884c\u3059\u308b</h3>\n",
|
||||
"<h3>Training or validation step</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u307e\u305f\u306f\u691c\u8a3c\u30b9\u30c6\u30c3\u30d7</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u3059\u308b\u8a2d\u5b9a\u306e\u8f9e\u66f8</p>\n",
|
||||
"<p>Calculate and cross entropy loss </p>\n": "<p>\u30af\u30ed\u30b9\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u640d\u5931\u306e\u8a08\u7b97\u3068\u30af\u30ed\u30b9\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u640d\u5931</p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u7cbe\u5ea6\u306e\u8a08\u7b97\u3068\u8a18\u9332</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u52fe\u914d\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Capacity factor to determine capacity of each model </p>\n": "<p>\u5404\u30e2\u30c7\u30eb\u306e\u5bb9\u91cf\u3092\u6c7a\u5b9a\u3059\u308b\u5bb9\u91cf\u4fc2\u6570</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30af\u30ea\u30a2</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u30af\u30ea\u30c3\u30d7\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Combined loss. The load balancing loss is multiplied by a coefficient <span translate=no>_^_0_^_</span> which is set to something small like <span translate=no>_^_1_^_</span>. </p>\n": "<p>\u8907\u5408\u640d\u5931\u3002\u8ca0\u8377\u5206\u6563\u640d\u5931\u306b\u306f\u3001<span translate=no>_^_0_^_</span>\u6b21\u306e\u3088\u3046\u306a\u5c0f\u3055\u306a\u5024\u306b\u8a2d\u5b9a\u3055\u308c\u305f\u4fc2\u6570\u304c\u4e57\u7b97\u3055\u308c\u307e\u3059</p>\u3002<span translate=no>_^_1_^_</span>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8131\u843d\u78ba\u7387</p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u6700\u7d42\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Fraction of tokens routed to each expert <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> is the count of tokens where the argmax of <span translate=no>_^_2_^_</span> is equal to <span translate=no>_^_3_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5404\u30a8\u30ad\u30b9\u30d1\u30fc\u30c8\u306b\u30eb\u30fc\u30c6\u30a3\u30f3\u30b0\u3055\u308c\u308b\u30c8\u30fc\u30af\u30f3\u306e\u5272\u5408\u306f\u3001argmax \u304c\u3068\u7b49\u3057\u3044\u30c8\u30fc\u30af\u30f3\u306e\u6570\u3067\u3059\u3002<span translate=no>_^_2_^_</span> <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210</p>\n",
|
||||
"<p>Get model outputs. </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u53d6\u5f97\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Initialize the subsequent mask </p>\n": "<p>\u5f8c\u7d9a\u306e\u30de\u30b9\u30af\u3092\u521d\u671f\u5316</p>\n",
|
||||
"<p>Initialize tracking indicators </p>\n": "<p>\u30c8\u30e9\u30c3\u30ad\u30f3\u30b0\u30a4\u30f3\u30b8\u30b1\u30fc\u30bf\u3092\u521d\u671f\u5316</p>\n",
|
||||
"<p>Load balancing coefficient </p>\n": "<p>\u8ca0\u8377\u5206\u6563\u4fc2\u6570</p>\n",
|
||||
"<p>Load balancing loss <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> is the loss for a single layer and here we are taking the sum of losses across all layers. </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u8ca0\u8377\u5206\u6563\u640d\u5931\u306f\u5358\u4e00\u30ec\u30a4\u30e4\u30fc\u306e\u640d\u5931\u3067\u3042\u308a\u3001\u3053\u3053\u3067\u306f\u3059\u3079\u3066\u306e\u30ec\u30a4\u30e4\u30fc\u306e\u640d\u5931\u306e\u5408\u8a08\u3092\u6c42\u3081\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u5404\u30a8\u30dd\u30c3\u30af\u306e\u6700\u5f8c\u306e\u30d0\u30c3\u30c1\u3067\u30e2\u30c7\u30eb\u30d1\u30e9\u30e1\u30fc\u30bf\u3068\u52fe\u914d\u3092\u8a18\u9332\u3057\u307e\u3059</p>\n",
|
||||
"<p>Mean routing probability <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e73\u5747\u30eb\u30fc\u30c6\u30a3\u30f3\u30b0\u78ba\u7387 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u30c7\u30fc\u30bf\u3092\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u6570</p>\n",
|
||||
"<p>Number of experts </p>\n": "<p>\u30a8\u30ad\u30b9\u30d1\u30fc\u30c8\u306e\u6570</p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN \u96a0\u308c\u30ec\u30a4\u30e4\u30fc\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u6570</p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u5909\u5727\u5668\u5c64\u306e\u6570</p>\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u5909\u5727\u5668\u306b\u901a\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u8ffd\u8de1\u3057\u305f\u30e1\u30c8\u30ea\u30af\u30b9\u3092\u4fdd\u5b58\u3059\u308b</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u6700\u9069\u5316\u306e\u4e00\u6b69\u3092\u8e0f\u307f\u51fa\u3059</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<p>Total number of tokens processed, <span translate=no>_^_0_^_</span>, in the current batch <span translate=no>_^_1_^_</span> </p>\n": "<p>\u73fe\u5728\u306e\u30d0\u30c3\u30c1\u3067\u51e6\u7406\u3055\u308c\u305f\u30c8\u30fc\u30af\u30f3\u306e\u7dcf\u6570 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Track stats </p>\n": "<p>\u30c8\u30e9\u30c3\u30af\u7d71\u8a08</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30e2\u30fc\u30c9\u6642\u306b\u30b0\u30ed\u30fc\u30d0\u30eb\u30b9\u30c6\u30c3\u30d7 (\u51e6\u7406\u3055\u308c\u305f\u30c8\u30fc\u30af\u30f3\u306e\u6570) \u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u30ad\u30e3\u30d7\u30c1\u30e3\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>Whether to drop tokens </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u3092\u30c9\u30ed\u30c3\u30d7\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>Whether to scale the chosen expert outputs by the routing probability </p>\n": "<p>\u9078\u629e\u3057\u305f\u30a8\u30ad\u30b9\u30d1\u30fc\u30c8\u30a2\u30a6\u30c8\u30d7\u30c3\u30c8\u3092\u30eb\u30fc\u30c6\u30a3\u30f3\u30b0\u78ba\u7387\u3067\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"Switch Transformer Experiment": "\u30b9\u30a4\u30c3\u30c1\u30c8\u30e9\u30f3\u30b9\u5b9f\u9a13",
|
||||
"This experiment trains a small switch transformer on tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u5c0f\u3055\u306a\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067\u5c0f\u3055\u306a\u30b9\u30a4\u30c3\u30c1\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"<h1>Switch Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a switch transformer.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/switch/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/353770ce177c11ecaa5fb74452424f46\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u0dc3\u0dca\u0dc0\u0dd2\u0da0\u0dca\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba\u0dc3\u0dca\u0dc0\u0dd2\u0da0\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0dbb\u0dca\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/switch/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/353770ce177c11ecaa5fb74452424f46\"> <span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This extends <a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a>.</p>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0dc0\u0dd2\u0dc4\u0dd2\u0daf\u0dda <a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a>. </p>\n<p>\u0d85\u0db4\u0dd2\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1 \u0dc0\u0dd2\u0da7 \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3 \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0d85\u0dad\u0dbb \u0d91\u0dba \u0d85\u0db0\u0dd2\u0d9a \u0dbd\u0dd9\u0dc3 \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0dc0\u0db1\u0dd4 \u0d87\u0dad</p>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Initialize the switch transformer</h3>\n": "<h3>\u0dc3\u0dca\u0dc0\u0dd2\u0da0\u0dca\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Training or validation step</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0dc4\u0ddd \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dd2\u0dba\u0dc0\u0dbb</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf\u0dba\u0dcf\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc1\u0db6\u0dca\u0daf\u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dca </p>\n",
|
||||
"<p>Calculate and cross entropy loss </p>\n": "<p>\u0d91\u0db1\u0dca\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd2\u0db1\u0dd0\u0dad\u0dd2\u0dc0\u0dd3\u0db8 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb \u0dc4\u0dbb\u0dc3\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba </p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Capacity factor to determine capacity of each model </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dda \u0db0\u0dcf\u0dbb\u0dd2\u0dad\u0dcf\u0dc0 \u0dad\u0dd3\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db0\u0dcf\u0dbb\u0dd2\u0dad\u0dcf \u0dc3\u0dcf\u0db0\u0d9a\u0dba </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d89\u0dc0\u0dad\u0dca </p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dca\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a </p>\n",
|
||||
"<p>Combined loss. The load balancing loss is multiplied by a coefficient <span translate=no>_^_0_^_</span> which is set to something small like <span translate=no>_^_1_^_</span>. </p>\n": "<p>\u0d92\u0d9a\u0dcf\u0db6\u0daf\u0dca\u0db0\u0d85\u0dbd\u0dcf\u0db7\u0dba. \u0db6\u0dbb \u0dad\u0dd4\u0dbd\u0db1\u0dba \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0dc0\u0dd0\u0db1\u0dd2 \u0d9a\u0dd4\u0da9\u0dcf \u0daf\u0dd9\u0dba\u0d9a\u0da7 \u0dc3\u0d9a\u0dc3\u0dcf <span translate=no>_^_0_^_</span> \u0d87\u0dad\u0dd2 \u0dc3\u0d82\u0d9c\u0dd4\u0dab\u0d9a\u0dba \u0db8\u0d9c\u0dd2\u0db1\u0dca \u0d9c\u0dd4\u0dab \u0d9a\u0dbb\u0db1\u0dd4 <span translate=no>_^_1_^_</span>\u0dbd\u0dd0\u0db6\u0dda. </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u0d85\u0dad\u0dc4\u0dd0\u0dbb\u0daf\u0dd0\u0db8\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0db1\u0dca\u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Fraction of tokens routed to each expert <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> is the count of tokens where the argmax of <span translate=no>_^_2_^_</span> is equal to <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc0\u0dd2\u0dc1\u0dda\u0dc2 expert \u0dba\u0dcf \u0dc0\u0dd9\u0dad \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dbd \u0db7\u0dcf\u0d9c\u0dba <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d86\u0d9c\u0db8\u0dca\u0db8\u0dd0\u0d9a\u0dca\u0dc3\u0dca \u0dc3\u0db8\u0dcf\u0db1 <span translate=no>_^_2_^_</span> \u0dc0\u0db1 \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1\u0dba\u0dba\u0dd2 <span translate=no>_^_3_^_</span>. </p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get model outputs. </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Initialize the subsequent mask </p>\n": "<p>\u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1\u0d86\u0dc0\u0dbb\u0dab \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize tracking indicators </p>\n": "<p>\u0dbd\u0dd4\u0dc4\u0dd4\u0db6\u0dd0\u0db3\u0dd3\u0db8\u0dda\u0daf\u0dbb\u0dca\u0dc1\u0d9a \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load balancing coefficient </p>\n": "<p>\u0db4\u0dd0\u0da7\u0dc0\u0dd4\u0db8\u0dca\u0dad\u0dd4\u0dbd\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0d82\u0d9c\u0dd4\u0dab\u0d9a\u0dba </p>\n",
|
||||
"<p>Load balancing loss <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> is the loss for a single layer and here we are taking the sum of losses across all layers. </p>\n": "<p>\u0db6\u0dbb\u0dad\u0dd4\u0dbd\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d85\u0dbd\u0dcf\u0db7\u0dba <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0dad\u0db1\u0dd2 \u0dad\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0db1 \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0db8\u0dd9\u0dc4\u0dd2\u0daf\u0dd3 \u0d85\u0db4\u0dd2 \u0dc3\u0dd2\u0dba\u0dbd\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0dc4\u0dbb\u0dc4\u0dcf \u0db4\u0dcf\u0da9\u0dd4 \u0d91\u0d9a\u0dad\u0dd4\u0dc0 \u0d9c\u0db1\u0dd2\u0db8\u0dd4. </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u0dc3\u0dd1\u0db8\u0dba\u0dd4\u0d9c\u0dbd\u0dba\u0d9a\u0db8 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0dc4 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Mean routing probability <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0db8\u0dcf\u0dbb\u0dca\u0d9c\u0d9c\u0dad \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dc0\u0dd9\u0dad \u0daf\u0dad\u0dca\u0dad \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of experts </p>\n": "<p>\u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0da5\u0dba\u0db1\u0dca\u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN\u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc3\u0dca\u0dae\u0dbb \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0dc4\u0dbb\u0dc4\u0dcf \u0d91\u0dba \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u0dbd\u0dd4\u0dc4\u0dd4\u0db6\u0dd0\u0db3\u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca </p>\n",
|
||||
"<p>Total number of tokens processed, <span translate=no>_^_0_^_</span>, in the current batch <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8 \u0dad\u0dd4\u0dc5 \u0dc3\u0dd0\u0d9a\u0dc3\u0dd6 \u0db8\u0dd4\u0dc5\u0dd4 \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1 <span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Track stats </p>\n": "<p>\u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0db1\u0dbd\u0dd4\u0dc4\u0dd4\u0db6\u0db3\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db4\u0dca\u0dbb\u0d9a\u0dcf\u0dbb\u0dba\u0dda\u0daf\u0dd3 \u0d9c\u0ddd\u0dbd\u0dd3\u0dba \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 (\u0dc3\u0dd0\u0d9a\u0dc3\u0dd6 \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1) </p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d9c\u0dca\u0dbb\u0dc4\u0dab\u0dba \u0d9a\u0dbb \u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether to drop tokens </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0d85\u0dad\u0dc4\u0dbb\u0dd2\u0db1\u0dca\u0db1 \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether to scale the chosen expert outputs by the routing probability </p>\n": "<p>\u0dbb\u0dc0\u0dd4\u0da7\u0dd2\u0db1\u0dca\u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 \u0d85\u0db1\u0dd4\u0dc0 \u0dad\u0ddd\u0dbb\u0dcf\u0d9c\u0dad\u0dca \u0dc0\u0dd2\u0dc1\u0dda\u0dc2 expert \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"Switch Transformer Experiment": "\u0dc3\u0dca\u0dc0\u0dd2\u0da0\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains a small switch transformer on tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db8\u0dad \u0d9a\u0dd4\u0da9\u0dcf \u0dc3\u0dca\u0dc0\u0dd2\u0da0\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"<h1>Switch Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a switch transformer.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/switch/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u5f00\u5173\u53d8\u538b\u5668\u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3\u5f00\u5173\u53d8\u538b\u5668\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/switch/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52a8\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This extends <a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a>.</p>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u5ef6\u4f38<a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a>\u4e86\u3002</p>\n<p>\u5f53\u6211\u4eec\u5f00\u59cb\u5b9e\u9a8c\u65f6\uff0c\u9ed8\u8ba4\u914d\u7f6e\u53ef\u4ee5\u800c\u4e14\u5c06\u4f1a\u88ab\u8986\u76d6</p>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u521d\u59cb\u5316\u81ea\u56de\u5f52\u6a21\u578b</h3>\n",
|
||||
"<h3>Initialize the switch transformer</h3>\n": "<h3>\u521d\u59cb\u5316\u5f00\u5173\u53d8\u538b\u5668</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u8fd0\u884c\u5b9e\u9a8c</h3>\n",
|
||||
"<h3>Training or validation step</h3>\n": "<h3>\u57f9\u8bad\u6216\u9a8c\u8bc1\u6b65\u9aa4</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u8981\u8986\u76d6\u7684\u914d\u7f6e\u5b57\u5178</p>\n",
|
||||
"<p>Calculate and cross entropy loss </p>\n": "<p>\u8ba1\u7b97\u548c\u4ea4\u53c9\u71b5\u635f\u5931</p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u8ba1\u7b97\u548c\u8bb0\u5f55\u7cbe\u5ea6</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"<p>Capacity factor to determine capacity of each model </p>\n": "<p>\u786e\u5b9a\u6bcf\u79cd\u578b\u53f7\u5bb9\u91cf\u7684\u5bb9\u91cf\u7cfb\u6570</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u6e05\u9664\u6e10\u53d8</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u526a\u8f91\u6e10\u53d8</p>\n",
|
||||
"<p>Combined loss. The load balancing loss is multiplied by a coefficient <span translate=no>_^_0_^_</span> which is set to something small like <span translate=no>_^_1_^_</span>. </p>\n": "<p>\u5408\u5e76\u4e8f\u635f\u3002\u8d1f\u8f7d\u5747\u8861\u635f\u5931\u4e58\u4ee5\u7cfb\u6570\uff0c<span translate=no>_^_0_^_</span>\u8be5\u7cfb\u6570\u8bbe\u7f6e\u4e3a\u7c7b\u4f3c\u7684\u5c0f\u503c<span translate=no>_^_1_^_</span>\u3002</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8f8d\u5b66\u6982\u7387</p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u6700\u540e\u4e00\u5c42</p>\n",
|
||||
"<p>Fraction of tokens routed to each expert <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> is the count of tokens where the argmax of <span translate=no>_^_2_^_</span> is equal to <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u53d1\u9001\u7ed9\u6bcf\u4e2a EA \u7684\u4ee3\u5e01\u7684\u5206\u6570<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u662f\u5176\u4e2d\u7684argmax\u7b49\u4e8e\u7684<span translate=no>_^_2_^_</span>\u4ee3\u5e01\u8ba1\u6570<span translate=no>_^_3_^_</span>\u3002</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u751f\u6210\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Get model outputs. </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u8f93\u51fa\u3002</p>\n",
|
||||
"<p>Initialize the subsequent mask </p>\n": "<p>\u521d\u59cb\u5316\u540e\u7eed\u63a9\u7801</p>\n",
|
||||
"<p>Initialize tracking indicators </p>\n": "<p>\u521d\u59cb\u5316\u8ddf\u8e2a\u6307\u793a\u5668</p>\n",
|
||||
"<p>Load balancing coefficient </p>\n": "<p>\u8d1f\u8f7d\u5e73\u8861\u7cfb\u6570</p>\n",
|
||||
"<p>Load balancing loss <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> is the loss for a single layer and here we are taking the sum of losses across all layers. </p>\n": "<p>\u8d1f\u8f7d\u5747\u8861\u635f\u5931<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u662f\u5355\u5c42\u7684\u635f\u5931\uff0c\u8fd9\u91cc\u6211\u4eec\u53d6\u7684\u662f\u6240\u6709\u5c42\u7684\u635f\u5931\u603b\u548c\u3002</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u8bb0\u5f55\u6bcf\u4e2a\u7eaa\u5143\u6700\u540e\u4e00\u6279\u7684\u6a21\u578b\u53c2\u6570\u548c\u68af\u5ea6</p>\n",
|
||||
"<p>Mean routing probability <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e73\u5747\u8def\u7531\u6982\u7387<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u5c06\u6570\u636e\u79fb\u52a8\u5230\u8bbe\u5907</p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u6ce8\u610f\u5934\u6570\u91cf</p>\n",
|
||||
"<p>Number of experts </p>\n": "<p>\u4e13\u5bb6\u4eba\u6570</p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN \u9690\u85cf\u5c42\u4e2d\u7684\u8981\u7d20\u6570\u91cf</p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u53d8\u538b\u5668\u5c42\u6570</p>\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u7528\u5b83\u7a7f\u8fc7\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u4fdd\u5b58\u8ddf\u8e2a\u7684\u6307\u6807</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u91c7\u53d6\u4f18\u5316\u5668\u6b65\u9aa4</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u6a21\u5757</p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Total number of tokens processed, <span translate=no>_^_0_^_</span>, in the current batch <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5f53\u524d\u6279\u6b21\u4e2d\u5df2\u5904\u7406\u7684\u4ee4\u724c\u603b\u6570<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Track stats </p>\n": "<p>\u8ffd\u8e2a\u7edf\u8ba1\u6570\u636e</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u8bad\u7ec3\u6a21\u578b</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u5728\u8bad\u7ec3\u6a21\u5f0f\u4e0b\u66f4\u65b0\u5168\u5c40\u6b65\u957f\uff08\u5904\u7406\u7684\u4ee4\u724c\u6570\uff09</p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u662f\u5426\u6355\u83b7\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>Whether to drop tokens </p>\n": "<p>\u662f\u5426\u4e22\u5f03\u4ee3\u5e01</p>\n",
|
||||
"<p>Whether to scale the chosen expert outputs by the routing probability </p>\n": "<p>\u662f\u5426\u6309\u8def\u7531\u6982\u7387\u7f29\u653e\u6240\u9009\u667a\u80fd\u4ea4\u6613\u8f93\u51fa</p>\n",
|
||||
"Switch Transformer Experiment": "\u5f00\u5173\u53d8\u538b\u5668\u5b9e\u9a8c",
|
||||
"This experiment trains a small switch transformer on tiny Shakespeare dataset.": "\u8fd9\u4e2a\u5b9e\u9a8c\u5728\u5fae\u5c0f\u7684\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u4e00\u4e2a\u5c0f\u578b\u5f00\u5173\u53d8\u538b\u5668\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/switch/index.html\">Switch Transformer</a></h1>\n<p>This is a miniature <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2101.03961\">Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity</a>. Our implementation only has a few million parameters and doesn't do model parallel distributed training. It does single GPU training, but we implement the concept of switching as described in the paper.</p>\n<p>The Switch Transformer uses different parameters for each token by switching among parameters based on the token. Therefore, only a fraction of parameters are chosen for each token. So you can have more parameters but less computational cost.</p>\n<p>The switching happens at the Position-wise Feedforward network (FFN) of each transformer block. Position-wise feedforward network consists of two sequentially fully connected layers. In switch transformer we have multiple FFNs (multiple experts), and we chose which one to use based on a router. The output is a set of probabilities for picking a FFN, and we pick the one with the highest probability and only evaluate that. So essentially the computational cost is the same as having a single FFN. In our implementation this doesn't parallelize well when you have many or large FFNs since it's all happening on a single GPU. In a distributed setup you would have each FFN (each very large) on a different device.</p>\n<p>The paper introduces another loss term to balance load among the experts (FFNs) and discusses dropping tokens when routing is not balanced.</p>\n<p>Here's <a href=\"experiment.html\">the training code</a> and a notebook for training a switch transformer on Tiny Shakespeare dataset. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/switch/index.html\">\u5f00\u5173\u53d8\u538b\u5668</a></h1>\n<p>\u8fd9\u662f\u7eb8\u8d28\u300a<a href=\"https://arxiv.org/abs/2101.03961\">\u5f00\u5173\u53d8\u5f62\u91d1\u521a\uff1a\u4ee5\u7b80\u5355\u9ad8\u6548\u7684\u7a00\u758f\u5ea6\u6269\u5c55\u5230\u4e07\u4ebf\u4e2a\u53c2\u6570\u6a21\u578b\u300b\u7684</a>\u5fae\u578b <a href=\"https://pytorch.org\">PyTorch</a> \u5b9e\u73b0\u3002\u6211\u4eec\u7684\u5b9e\u73b0\u53ea\u6709\u51e0\u767e\u4e07\u4e2a\u53c2\u6570\uff0c\u4e0d\u5bf9\u5e76\u884c\u5206\u5e03\u5f0f\u8bad\u7ec3\u8fdb\u884c\u5efa\u6a21\u3002\u5b83\u8fdb\u884c\u5355\u4e2a GPU \u8bad\u7ec3\uff0c\u4f46\u6211\u4eec\u5b9e\u73b0\u4e86\u8bba\u6587\u4e2d\u63cf\u8ff0\u7684\u5207\u6362\u6982\u5ff5\u3002</p>\n<p>Switch Transformer \u901a\u8fc7\u6839\u636e\u4ee4\u724c\u5728\u53c2\u6570\u4e4b\u95f4\u5207\u6362\uff0c\u4e3a\u6bcf\u4e2a\u4ee4\u724c\u4f7f\u7528\u4e0d\u540c\u7684\u53c2\u6570\u3002\u56e0\u6b64\uff0c\u53ea\u4e3a\u6bcf\u4e2a\u4ee3\u5e01\u9009\u62e9\u4e86\u4e00\u5c0f\u90e8\u5206\u53c2\u6570\u3002\u56e0\u6b64\uff0c\u60a8\u53ef\u4ee5\u62e5\u6709\u66f4\u591a\u53c2\u6570\uff0c\u4f46\u8ba1\u7b97\u6210\u672c\u66f4\u4f4e\u3002</p>\n<p>\u5207\u6362\u53d1\u751f\u5728\u6bcf\u4e2a\u53d8\u538b\u5668\u6a21\u5757\u7684\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc (FFN) \u4e0a\u3002\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc\u7531\u4e24\u4e2a\u6309\u987a\u5e8f\u5b8c\u5168\u8fde\u63a5\u7684\u5c42\u7ec4\u6210\u3002\u5728\u4ea4\u6362\u673a\u53d8\u538b\u5668\u4e2d\uff0c\u6211\u4eec\u6709\u591a\u4e2a FFN\uff08\u591a\u4f4d\u4e13\u5bb6\uff09\uff0c\u6211\u4eec\u6839\u636e\u8def\u7531\u5668\u9009\u62e9\u4f7f\u7528\u54ea\u4e00\u4e2a\u3002\u8f93\u51fa\u662f\u4e00\u7ec4\u7528\u4e8e\u9009\u62e9 FFN \u7684\u6982\u7387\uff0c\u6211\u4eec\u9009\u62e9\u6982\u7387\u6700\u9ad8\u7684\u6982\u7387\uff0c\u7136\u540e\u4ec5\u5bf9\u5176\u8fdb\u884c\u8bc4\u4f30\u3002\u56e0\u6b64\uff0c\u4ece\u672c\u8d28\u4e0a\u8bb2\uff0c\u8ba1\u7b97\u6210\u672c\u4e0e\u62e5\u6709\u5355\u4e2a FFN \u76f8\u540c\u3002\u5728\u6211\u4eec\u7684\u5b9e\u73b0\u4e2d\uff0c\u5f53\u4f60\u6709\u8bb8\u591a\u6216\u5927\u578b FFN \u65f6\uff0c\u8fd9\u79cd\u5e76\u884c\u5316\u6548\u679c\u4e0d\u4f73\uff0c\u56e0\u4e3a\u8fd9\u4e00\u5207\u90fd\u53d1\u751f\u5728\u5355\u4e2a GPU \u4e0a\u3002\u5728\u5206\u5e03\u5f0f\u8bbe\u7f6e\u4e2d\uff0c\u4f60\u4f1a\u5c06\u6bcf\u4e2a FFN\uff08\u6bcf\u4e2a\u90fd\u5f88\u5927\uff09\u653e\u5728\u4e0d\u540c\u7684\u8bbe\u5907\u4e0a\u3002</p>\n<p>\u672c\u6587\u5f15\u5165\u4e86\u53e6\u4e00\u4e2a\u635f\u5931\u672f\u8bed\u6765\u5e73\u8861\u4e13\u5bb6\uff08FFN\uff09\u4e4b\u95f4\u7684\u8d1f\u8f7d\uff0c\u5e76\u8ba8\u8bba\u4e86\u8def\u7531\u4e0d\u5e73\u8861\u65f6\u4e22\u5f03\u4ee3\u5e01\u7684\u95ee\u9898\u3002</p>\n<p>\u8fd9\u662f<a href=\"experiment.html\">\u8bad\u7ec3\u4ee3\u7801\u548c\u4e00\u672c</a>\u7528\u4e8e\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u5f00\u5173\u53d8\u538b\u5668\u7684\u7b14\u8bb0\u672c\u3002</p>\n",
|
||||
"Switch Transformer": "\u5f00\u5173\u53d8\u538b\u5668"
|
||||
}
|
||||
Reference in New Issue
Block a user