chore: import upstream snapshot with attribution

2026-07-13 12:19:01 +08:00
commit 3b90d1192f
2172 changed files with 594509 additions and 0 deletions
@@ -0,0 +1,34 @@
+{
+ "<h1>Pay Attention to MLPs (gMLP)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a gMLP model based autoregressive model.</p>\n": "<h1>MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044</h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.08050\">MLP\u306b\u6ce8\u610f\u3057\u3066</a>\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a>\u3002</p>\n<p><strong>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30b2\u30fc\u30c6\u30a3\u30f3\u30b0\u3092\u5099\u3048\u305f\u591a\u5c64\u30d1\u30fc\u30bb\u30d7\u30c8\u30ed\u30f3\uff08MLP\uff09\u30d9\u30fc\u30b9\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\uff08GMLP\u3068\u540d\u4ed8\u3051\u3089\u308c\u3066\u3044\u307e\u3059\uff09\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002</strong><span translate=no>_^_0_^_</span><em>gMLP</em> \u30d6\u30ed\u30c3\u30af\u306e\u30b9\u30bf\u30c3\u30af\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059</p>\u3002\n<p><a href=\"experiment.html\">gMLP\u30e2\u30c7\u30eb\u30d9\u30fc\u30b9\u306e\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059</a>\u3002</p>\n",
+ "<h2>Spatial Gating Unit</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a linear transformation along the sequence dimension, and <span translate=no>_^_2_^_</span> is element-wise multiplication. <span translate=no>_^_3_^_</span> is split into to parts of equal size <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> along the channel dimension (embedding dimension).</p>\n": "<h2>\u7a7a\u9593\u30b2\u30fc\u30c8\u30e6\u30cb\u30c3\u30c8</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>\u3053\u3053\u3067<span translate=no>_^_1_^_</span>\u3001\u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u6b21\u5143\u306b\u6cbf\u3063\u305f\u7dda\u5f62\u5909\u63db\u3067\u3001<span translate=no>_^_2_^_</span>\u306f\u8981\u7d20\u5358\u4f4d\u306e\u4e57\u7b97\u3067\u3059\u3002<span translate=no>_^_3_^_</span>\u30c1\u30e3\u30cd\u30eb\u5bf8\u6cd5\uff08\u57cb\u3081\u8fbc\u307f\u5bf8\u6cd5\uff09<span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span>\u306b\u6cbf\u3063\u3066\u540c\u3058\u30b5\u30a4\u30ba\u306e2\u3064\u306e\u90e8\u5206\u306b\u5206\u5272\u3055\u308c\u307e\u3059</p>\u3002\n",
+ "<h2>gMLP Block</h2>\n<p>Each block does the following transformations to input embeddings <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the sequence length and <span translate=no>_^_2_^_</span> is the dimensionality of the embeddings:</p>\n<span translate=no>_^_3_^_</span><p>where <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are learnable projection weights. <span translate=no>_^_6_^_</span> is the Spacial Gating Unit defined below. Output dimensionality of <span translate=no>_^_7_^_</span> will be half of <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span> is an activation function such as <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLU</a>.</p>\n": "<h2>GmLP \u30d6\u30ed\u30c3\u30af</h2>\n<p>\u5404\u30d6\u30ed\u30c3\u30af\u306f\u3001\u5165\u529b\u57cb\u3081\u8fbc\u307f\u306b\u5bfe\u3057\u3066\u6b21\u306e\u5909\u63db\u3092\u884c\u3044\u307e\u3059\u3002<span translate=no>_^_0_^_</span>\u3053\u3053\u3067\u3001\u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u306f\u57cb\u3081\u8fbc\u307f\u306e\u6b21\u5143\u3067\u3059\u3002</p>\n<span translate=no>_^_3_^_</span><p><span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span>\u5b66\u7fd2\u53ef\u80fd\u306a\u6295\u5f71\u91cd\u307f\u306e\u4f4d\u7f6e\u3068\u4f4d\u7f6e<span translate=no>_^_6_^_</span>\u306f\u4ee5\u4e0b\u306b\u5b9a\u7fa9\u3059\u308b\u30b9\u30da\u30fc\u30b7\u30e3\u30eb\u30fb\u30b2\u30fc\u30c6\u30a3\u30f3\u30b0\u30fb\u30e6\u30cb\u30c3\u30c8\u3067\u3059\u3002<span translate=no>_^_7_^_</span>\u306e\u51fa\u529b\u6b21\u5143\u306f\u306e\u534a\u5206\u306b\u306a\u308a\u307e\u3059\u3002<span translate=no>_^_8_^_</span><span translate=no>_^_9_^_</span><a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">\u306fGelU\u306e\u3088\u3046\u306a\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u95a2\u6570\u3067\u3059</a></p>\u3002\n",
+ "<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
+ "<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span>. The batch dimension should be of size <span translate=no>_^_2_^_</span> because this implementation supports only same mask for all samples in the batch. </p>\n": "<p><span translate=no>_^_0_^_</span>\u5f62\u304c\u3042\u308a\u307e\u3059<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u3053\u306e\u5b9f\u88c5\u3067\u306f\u30d0\u30c3\u30c1\u5185\u306e\u3059\u3079\u3066\u306e\u30b5\u30f3\u30d7\u30eb\u306b\u5bfe\u3057\u3066\u540c\u3058\u30de\u30b9\u30af\u3057\u304b\u30b5\u30dd\u30fc\u30c8\u3057\u306a\u3044\u305f\u3081\u3001\u30d0\u30c3\u30c1\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u306f\u30b5\u30a4\u30ba\u306b\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002</p>\n",
+ "<p>Activation function <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u6a5f\u80fd <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Add the shortcut connection </p>\n": "<p>\u30b7\u30e7\u30fc\u30c8\u30ab\u30c3\u30c8\u63a5\u7d9a\u3092\u8ffd\u52a0\u3059\u308b</p>\n",
+ "<p>Apply mask to the weights.</p>\n<p>If <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> then <span translate=no>_^_2_^_</span> will not get any information from token <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u30a6\u30a7\u30a4\u30c8\u306b\u30de\u30b9\u30af\u3092\u304b\u3051\u307e\u3059\u3002</p>\n<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3082\u3057\u305d\u3046\u306a\u3089<span translate=no>_^_2_^_</span>\u3001\u30c8\u30fc\u30af\u30f3\u304b\u3089\u60c5\u5831\u3092\u53d6\u5f97\u3059\u308b\u3053\u3068\u306f\u306a\u3044<span translate=no>_^_3_^_</span>\u3002</p>\n",
+ "<p>Check mask </p>\n": "<p>\u30c1\u30a7\u30c3\u30af\u30de\u30b9\u30af</p>\n",
+ "<p>Embedding size (required by <a href=\"../models.html#Encoder\">Encoder</a>. We use the encoder module from transformer architecture and plug <em>gMLP</em> block as a replacement for the <a href=\"../models.html#Encoder\">Transformer Layer</a>. </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba (<a href=\"../models.html#Encoder\">\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u5fc5\u8981</a>)</p><a href=\"../models.html#Encoder\">\u30c8\u30e9\u30f3\u30b9\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u30a8\u30f3\u30b3\u30fc\u30c0\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u4f7f\u7528\u3057\u3001<em>\u30c8\u30e9\u30f3\u30b9\u30ec\u30a4\u30e4\u306e\u4ee3\u308f\u308a\u306bgMLP\u30d6\u30ed\u30c3\u30af\u3092\u30d7\u30e9\u30b0\u3057\u307e\u3059</em>\u3002</a>\n",
+ "<p>Final projection <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u7d42\u6295\u5f71 <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Get sequence length </p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3092\u53d6\u5f97</p>\n",
+ "<p>Get the weight matrix; truncate if larger than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30a6\u30a7\u30a4\u30c8\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u3092\u53d6\u5f97\u3002\u3053\u308c\u3088\u308a\u5927\u304d\u3044\u5834\u5408\u306f\u5207\u308a\u6368\u3066\u308b <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Here we only support the same mask for all samples </p>\n": "<p>\u3053\u3053\u3067\u306f\u3001\u3059\u3079\u3066\u306e\u30b5\u30f3\u30d7\u30eb\u3067\u540c\u3058\u30de\u30b9\u30af\u306e\u307f\u3092\u30b5\u30dd\u30fc\u30c8\u3057\u3066\u3044\u307e\u3059</p>\n",
+ "<p>Keep a copy for shortcut connection </p>\n": "<p>\u30b7\u30e7\u30fc\u30c8\u30ab\u30c3\u30c8\u63a5\u7d9a\u7528\u306b\u30b3\u30d4\u30fc\u3092\u4fdd\u5b58</p>\n",
+ "<p>Normalization layer before applying <span translate=no>_^_0_^_</span> </p>\n": "<p>\u9069\u7528\u524d\u306e\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Normalization layer fro Pre-Norm </p>\n": "<p>\u30d7\u30ec\u30ce\u30eb\u30e0\u306e\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc</p>\n",
+ "<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30ce\u30fc\u30de\u30e9\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Normalize <span translate=no>_^_0_^_</span> before <span translate=no>_^_1_^_</span> </p>\n": "<p>\u524d\u306b\u30ce\u30fc\u30de\u30e9\u30a4\u30ba <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
+ "<p>Projection and activation <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d7\u30ed\u30b8\u30a7\u30af\u30b7\u30e7\u30f3\u3068\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3 <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Projection layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u306e\u6295\u5f71\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Remove the batch dimension </p>\n": "<p>\u30d0\u30c3\u30c1\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u3092\u524a\u9664\u3059\u308b</p>\n",
+ "<p>Spacial Gating Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7a7a\u9593\u30b2\u30fc\u30c8\u30e6\u30cb\u30c3\u30c8 <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Split <span translate=no>_^_0_^_</span> into <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3068\u306b\u5206\u5272 <span translate=no>_^_2_^_</span></p>\n",
+ "<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span>.</p>\n<p>The paper notes that it&#x27;s important to initialize weights to small values and the bias to <span translate=no>_^_2_^_</span>, so that during the initial training <span translate=no>_^_3_^_</span> is close to identity (apart from the split). </p>\n": "<p>\u91cd\u91cf <span translate=no>_^_0_^_</span> (\u30a4\u30f3<span translate=no>_^_1_^_</span>)</p>\n<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u91cd\u307f\u3092\u5c0f\u3055\u3044\u5024\u306b\u521d\u671f\u5316\u3057\u3001\u30d0\u30a4\u30a2\u30b9\u3092\u306b\u521d\u671f\u5316\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3042\u308b\u3068\u8ff0\u3079\u3066\u3044\u307e\u3059\u3002\u305d\u3046\u3059\u308c\u3070<span translate=no>_^_2_^_</span>\u3001\u6700\u521d\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\uff08<span translate=no>_^_3_^_</span>\u5206\u5272\u306f\u5225\u3068\u3057\u3066\uff09\u540c\u4e00\u306b\u8fd1\u3044\u3082\u306e\u306b\u306a\u308a\u307e\u3059\u3002</p>\n",
+ "<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span></p>\n<p>The paper notes that it&#x27;s important to initialize bias to <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u91cd\u91cf (<span translate=no>_^_0_^_</span>\u30a4\u30f3) <span translate=no>_^_1_^_</span></p>\n<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30d0\u30a4\u30a2\u30b9\u3092\u306b\u521d\u671f\u5316\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3060\u3068\u6307\u6458\u3057\u3066\u3044\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the dimensionality (<span translate=no>_^_1_^_</span>) of <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the dimensionality of <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the length of the token sequence (<span translate=no>_^_6_^_</span>)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u306e\u6b21\u5143 () <span translate=no>_^_1_^_</span> <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u306e\u6b21\u5143\u3067\u3059 <span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u306f\u30c8\u30fc\u30af\u30f3\u30fb\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055 (<span translate=no>_^_6_^_</span>)</li></ul>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the dimensionality of <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the sequence length</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306e\u6b21\u5143\u3067\u3059 <span translate=no>_^_1_^_</span></li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3067\u3059</li></ul>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the input <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens  among each other. The last dimension of size <span translate=no>_^_5_^_</span> is the batch, which we have in other transformer  implementations and was left for compatibility.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5f62\u72b6\u306e\u5165\u529b\u3067\u3059 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>is \u306f\u3001<span translate=no>_^_4_^_</span>\u30c8\u30fc\u30af\u30f3\u540c\u58eb\u306e\u53ef\u8996\u6027\u3092\u5236\u5fa1\u3059\u308b\u30d6\u30fc\u30ea\u30a2\u30f3\u30de\u30b9\u30af\u3067\u3059\u3002<span translate=no>_^_5_^_</span>\u30b5\u30a4\u30ba\u306e\u6700\u5f8c\u306e\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u306f\u30d0\u30c3\u30c1\u3067\u3059\u3002\u3053\u308c\u306f\u4ed6\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u5b9f\u88c5\u306b\u3082\u3042\u308a\u307e\u3059\u304c\u3001\u4e92\u63db\u6027\u306e\u305f\u3081\u306b\u6b8b\u3055\u308c\u3066\u3044\u307e\u3059</li></ul>\u3002\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the input embedding tensor <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens  among each other.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5f62\u72b6\u306e\u5165\u529b\u57cb\u3081\u8fbc\u307f\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_2_^_</span></li>\n</ul><li><span translate=no>_^_3_^_</span>\u306f\u3001<span translate=no>_^_4_^_</span>\u30c8\u30fc\u30af\u30f3\u540c\u58eb\u306e\u53ef\u8996\u6027\u3092\u5236\u5fa1\u3059\u308b\u30d6\u30fc\u30ea\u30a2\u30f3\u30b7\u30a7\u30a4\u30d7\u30de\u30b9\u30af\u3067\u3059\u3002</li>\n",
+ "Pay Attention to MLPs (gMLP)": "MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044",
+ "This is an annotated implementation/tutorial of Pay Attention to MLPs (gMLP) in PyTorch.": "\u3053\u308c\u306f PyTorch \u306e\u300cMLP\uff08GMLP\uff09\u306b\u6ce8\u610f\u300d\u306e\u6ce8\u91c8\u4ed8\u304d\u306e\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002"
+}
@@ -0,0 +1,34 @@
+{
+ "<h1>Pay Attention to MLPs (gMLP)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a gMLP model based autoregressive model.</p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>MLPs(GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/2105.08050\">MLPs \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a> . </p>\n<p>\u0db8\u0dd9\u0db8\u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0db6\u0dc4\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0db4\u0dbb\u0dca\u0dc3\u0dd9\u0db4\u0dca\u0da7\u0dca\u0dbb\u0ddd\u0db1\u0dca (\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3) \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0d9a\u0dca \u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0d9f \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0d85\u0dad\u0dbb \u0d92\u0dc0\u0dcf <strong>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3</strong>\u0dbd\u0dd9\u0dc3 \u0db1\u0db8\u0dca \u0d9a\u0dbb\u0dba\u0dd2. \u0d91\u0dba <span translate=no>_^_0_^_</span> <em>GMLP</em> \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dad\u0ddc\u0d9c\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0db1\u0dca\u0dc0\u0dd2\u0dad \u0dc0\u0dda. </p>\n<p>GMLP\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf <a href=\"experiment.html\">\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dda\u0dad\u0dba</a> \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_1_^_</span></a></p>\n",
+ "<h2>Spatial Gating Unit</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a linear transformation along the sequence dimension, and <span translate=no>_^_2_^_</span> is element-wise multiplication. <span translate=no>_^_3_^_</span> is split into to parts of equal size <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> along the channel dimension (embedding dimension).</p>\n": "<h2>\u0d85\u0dc0\u0d9a\u0dcf\u0dc1\u0dd3\u0dba\u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0d92\u0d9a\u0d9a\u0dba</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0db8\u0dcf\u0db1\u0dba \u0d94\u0dc3\u0dca\u0dc3\u0dda \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0d9a\u0dca \u0dc0\u0db1 \u0d85\u0dad\u0dbb <span translate=no>_^_2_^_</span> \u0d91\u0dba \u0db8\u0dd6\u0dbd\u0daf\u0dca\u0dbb\u0dc0\u0dca\u0dba \u0d85\u0db1\u0dd4\u0dc0 \u0d9c\u0dd4\u0dab \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2. <span translate=no>_^_1_^_</span> <span translate=no>_^_3_^_</span> \u0dc3\u0db8\u0dcf\u0db1 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dda \u0d9a\u0ddc\u0da7\u0dc3\u0dca \u0dc0\u0dbd\u0da7 <span translate=no>_^_4_^_</span> \u0dc3\u0dc4 \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0db8\u0dcf\u0db1\u0dba <span translate=no>_^_5_^_</span> \u0d94\u0dc3\u0dca\u0dc3\u0dda (\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db8\u0dcf\u0db1\u0dba) \u0db6\u0dd9\u0daf\u0dcf \u0d87\u0dad. </p>\n",
+ "<h2>gMLP Block</h2>\n<p>Each block does the following transformations to input embeddings <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the sequence length and <span translate=no>_^_2_^_</span> is the dimensionality of the embeddings:</p>\n<span translate=no>_^_3_^_</span><p>where <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are learnable projection weights. <span translate=no>_^_6_^_</span> is the Spacial Gating Unit defined below. Output dimensionality of <span translate=no>_^_7_^_</span> will be half of <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span> is an activation function such as <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLU</a>.</p>\n": "<h2>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3\u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca</h2>\n<p>\u0dc3\u0dd1\u0db8\u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca \u0d91\u0d9a\u0d9a\u0dca\u0db8 \u0d86\u0daf\u0dcf\u0db1 <span translate=no>_^_0_^_</span> \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dc4\u0dad \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd9\u0db1 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0db1\u0dca \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0daf\u0dd2\u0d9c \u0dc3\u0dc4 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca <span translate=no>_^_2_^_</span> \u0dc0\u0dda: <span translate=no>_^_1_^_</span> </p>\n<span translate=no>_^_3_^_</span><p>\u0d89\u0d9c\u0dd9\u0db1 <span translate=no>_^_5_^_</span> \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2 \u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0dab \u0db6\u0dbb \u0d9a\u0ddc\u0dc4\u0dda\u0daf <span translate=no>_^_4_^_</span> \u0dc3\u0dc4 \u0d87\u0dad. <span translate=no>_^_6_^_</span> \u0db4\u0dc4\u0dad \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0d85\u0db7\u0dca\u0dba\u0dc0\u0d9a\u0dcf\u0dc1 \u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0d92\u0d9a\u0d9a\u0dba \u0dc0\u0dda. \u0db1\u0dd2\u0db8\u0dd0\u0dc0\u0dd4\u0db8\u0dca \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d85\u0da9\u0d9a\u0dca <span translate=no>_^_7_^_</span> \u0dc0\u0db1\u0dd4 \u0d87\u0dad <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span> <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLu</a>\u0dc0\u0dd0\u0db1\u0dd2 \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0d9a\u0dd2. </p>\n",
+ "<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
+ "<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span>. The batch dimension should be of size <span translate=no>_^_2_^_</span> because this implementation supports only same mask for all samples in the batch. </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba \u0d87\u0dad <span translate=no>_^_1_^_</span>. \u0db8\u0dd9\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0db8 \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0db4\u0db8\u0dab\u0d9a\u0dca \u0dc3\u0dc4\u0dcf\u0dba \u0dc0\u0db1 <span translate=no>_^_2_^_</span> \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db8\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dd9\u0db1\u0dca \u0dba\u0dd4\u0d9a\u0dca\u0dad \u0dc0\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4\u0dba. </p>\n",
+ "<p>Activation function <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Add the shortcut connection </p>\n": "<p>\u0d9a\u0dd9\u0da7\u0dd2\u0db8\u0d82\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dad\u0dcf\u0dc0\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Apply mask to the weights.</p>\n<p>If <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> then <span translate=no>_^_2_^_</span> will not get any information from token <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u0db4\u0da9\u0dd2\u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0dc0\u0dbb\u0dab \u0dba\u0ddc\u0daf\u0db1\u0dca\u0db1. </p>\n<p>\u0d91\u0dc3\u0dda <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u0db1\u0db8\u0dca <span translate=no>_^_2_^_</span> \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd9\u0db1\u0dca \u0d9a\u0dd2\u0dc3\u0dd2\u0daf\u0dd4 \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0d9a\u0dca \u0dbd\u0db6\u0dcf <span translate=no>_^_3_^_</span>\u0db1\u0ddc\u0d9c\u0db1\u0dd3. </p>\n",
+ "<p>Check mask </p>\n": "<p>\u0d86\u0dc0\u0dbb\u0dab\u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Embedding size (required by <a href=\"../models.html#Encoder\">Encoder</a>. We use the encoder module from transformer architecture and plug <em>gMLP</em> block as a replacement for the <a href=\"../models.html#Encoder\">Transformer Layer</a>. </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba ( <a href=\"../models.html#Encoder\">\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a>\u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dda. \u0d85\u0db4\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0dd9\u0db1\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"../models.html#Encoder\">\u0dc3\u0dca\u0dae\u0dbb\u0dba</a>\u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7 \u0d86\u0daf\u0dda\u0dc1\u0d9a\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 <em>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3</em> \u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca \u0db4\u0dca\u0dbd\u0d9c\u0dca \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
+ "<p>Final projection <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Get sequence length </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0daf\u0dd2\u0d9c \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Get the weight matrix; truncate if larger than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db6\u0dbb\u0d85\u0db1\u0dd4\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1; \u0dc0\u0da9\u0dcf \u0dc0\u0dd2\u0dc1\u0dcf\u0dbd \u0db1\u0db8\u0dca \u0da7\u0dca\u0dbb\u0db1\u0dca\u0d9a\u0dda\u0da7\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Here we only support the same mask for all samples </p>\n": "<p>\u0db8\u0dd9\u0db1\u0dca\u0db1\u0d85\u0db4\u0dd2 \u0dc3\u0dc4\u0dcf\u0dba \u0daf\u0dd9\u0db1\u0dca\u0db1\u0dda \u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0db8 \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0db4\u0db8\u0dab\u0dd2 </p>\n",
+ "<p>Keep a copy for shortcut connection </p>\n": "<p>\u0d9a\u0dd9\u0da7\u0dd2\u0db8\u0d82\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dad\u0dcf\u0dc0\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd2\u0da7\u0db4\u0dad\u0d9a\u0dca \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Normalization layer before applying <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dba\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db4\u0dd9\u0dbb \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Normalization layer fro Pre-Norm </p>\n": "<p>\u0db4\u0dd9\u0dbb-\u0dc0\u0dd0\u0da9\u0dd9\u0db1\u0dca\u0db1\u0dda\u0dc3\u0dd2\u0da7 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
+ "<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u200d\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Normalize <span translate=no>_^_0_^_</span> before <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> \u0db4\u0dd9\u0dbb \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba <span translate=no>_^_1_^_</span> </p>\n",
+ "<p>Projection and activation <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0dab\u0dba\u0dc3\u0dc4 \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Projection layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0db3\u0dc4\u0dcf\u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Remove the batch dimension </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db8\u0dcf\u0db1\u0dba \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Spacial Gating Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0db7\u0dca\u0dba\u0dc0\u0d9a\u0dcf\u0dc1\u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0d92\u0d9a\u0d9a\u0dba <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Split <span translate=no>_^_0_^_</span> into <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> \u0db6\u0dd9\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> \u0dc3\u0dc4 <span translate=no>_^_2_^_</span> </p>\n",
+ "<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span>.</p>\n<p>The paper notes that it&#x27;s important to initialize weights to small values and the bias to <span translate=no>_^_2_^_</span>, so that during the initial training <span translate=no>_^_3_^_</span> is close to identity (apart from the split). </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc3\u0dd2\u0dbb\u0dd4\u0dbb\u0dda \u0db6\u0dbb <span translate=no>_^_1_^_</span>. </p>\n<p>\u0d9a\u0dd4\u0da9\u0dcf\u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc4\u0dcf \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0db6\u0dbb \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc0\u0dd0\u0daf\u0d9c\u0dad\u0dca \u0db6\u0dc0 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0d9a\u0dbb\u0dba\u0dd2 <span translate=no>_^_2_^_</span>, \u0d91\u0dc0\u0dd2\u0da7 \u0db8\u0dd6\u0dbd\u0dd2\u0d9a \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dad\u0dd4\u0dc5 \u0d85\u0db1\u0db1\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba\u0da7 \u0dc3\u0db8\u0dd3\u0db4 <span translate=no>_^_3_^_</span> \u0dc0\u0dda (\u0db7\u0dda\u0daf\u0dba \u0dc4\u0dd0\u0dbb\u0dd4\u0dab\u0dd4 \u0dc0\u0dd2\u0da7). </p>\n",
+ "<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span></p>\n<p>The paper notes that it&#x27;s important to initialize bias to <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u0dc3\u0dd2\u0dbb\u0dd4\u0dbb\u0dda\u0db6\u0dbb <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n<p>\u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3\u0dc0\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc0\u0dd0\u0daf\u0d9c\u0dad\u0dca \u0db6\u0dc0 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc3\u0da7\u0dc4\u0db1\u0dca <span translate=no>_^_2_^_</span>\u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the dimensionality (<span translate=no>_^_1_^_</span>) of <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the dimensionality of <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the length of the token sequence (<span translate=no>_^_6_^_</span>)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca (<span translate=no>_^_1_^_</span>) <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> \u0dc4\u0dd2 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc0\u0dda <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda \u0daf\u0dd2\u0d9c (<span translate=no>_^_6_^_</span>)</li></ul>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the dimensionality of <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the sequence length</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd2 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc0\u0dda <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba \u0daf\u0dd2\u0d9c \u0dc0\u0dda</li></ul>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the input <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens  among each other. The last dimension of size <span translate=no>_^_5_^_</span> is the batch, which we have in other transformer  implementations and was left for compatibility.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda <span translate=no>_^_1_^_</span> \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dc0\u0dda <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> \u0dba\u0db1\u0dd4 \u0d91\u0d9a\u0dd2\u0db1\u0dd9\u0d9a\u0dcf \u0d85\u0dad\u0dbb \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dbd \u0daf\u0dd8\u0dc1\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db4\u0dcf\u0dbd\u0db1\u0dba <span translate=no>_^_4_^_</span> \u0d9a\u0dbb\u0db1 \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0db6\u0dd6\u0dbd\u0dd2\u0dba\u0db1\u0dca \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dd2. \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dda \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db8\u0dcf\u0db1\u0dba <span translate=no>_^_5_^_</span> \u0dc0\u0db1\u0dca\u0db1\u0dda \u0dc0\u0dd9\u0db1\u0dad\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0dc0\u0dbd\u0daf\u0dd3 \u0d85\u0db4 \u0dc3\u0dad\u0dd4\u0dc0 \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0d85\u0db1\u0dd4\u0d9a\u0dd6\u0dbd\u0dad\u0dcf\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0dc0 \u0d87\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dba\u0dd2. </li></ul>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the input embedding tensor <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens  among each other.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d86\u0daf\u0dcf\u0db1 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 <span translate=no>_^_1_^_</span> \u0dc0\u0dda <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> \u0dba\u0db1\u0dd4 \u0d91\u0d9a\u0dd2\u0db1\u0dd9\u0d9a\u0dcf \u0d85\u0dad\u0dbb \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dbd \u0daf\u0dd8\u0dc1\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db4\u0dcf\u0dbd\u0db1\u0dba <span translate=no>_^_4_^_</span> \u0d9a\u0dbb\u0db1 \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0db6\u0dd6\u0dbd\u0dd2\u0dba\u0db1\u0dca \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dd2. </li></ul>\n",
+ "Pay Attention to MLPs (gMLP)": "MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1",
+ "This is an annotated implementation/tutorial of Pay Attention to MLPs (gMLP) in PyTorch.": "\u0db8\u0dd9\u0dba PyTorch \u0dc4\u0dd2 MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc0\u0dd2\u0db1\u0dba\u0d9a\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba\u0d9a\u0dd2."
+}
@@ -0,0 +1,34 @@
+{
+ "<h1>Pay Attention to MLPs (gMLP)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a gMLP model based autoregressive model.</p>\n": "<h1>\u6ce8\u610f MLP (GmLP)</h1>\n<p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u5bf9\u300a<a href=\"https://arxiv.org/abs/2105.08050\">\u6ce8\u610f MLP\u300b\u4e00\u6587\u7684</a>\u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u4ecb\u7ecd\u4e86\u4e00\u79cd\u57fa\u4e8e\u591a\u5c42\u611f\u77e5\u5668\uff08MLP\uff09\u7684\u5e26\u6709\u95e8\u63a7\u7684\u67b6\u6784\uff0c\u4ed6\u4eec\u5c06\u5176\u547d\u540d\u4e3a <strong>gmLP</strong>\u3002\u5b83\u7531\u4e00\u5806<span translate=no>_^_0_^_</span> <em>gmLP</em> \u5757\u7ec4\u6210\u3002</p>\n<p>\u8fd9\u662f\u57fa<a href=\"experiment.html\">\u4e8e GmLP \u6a21\u578b\u7684\u81ea\u56de\u5f52\u6a21\u578b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
+ "<h2>Spatial Gating Unit</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a linear transformation along the sequence dimension, and <span translate=no>_^_2_^_</span> is element-wise multiplication. <span translate=no>_^_3_^_</span> is split into to parts of equal size <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> along the channel dimension (embedding dimension).</p>\n": "<h2>\u7a7a\u95f4\u95e8\u63a7\u5355\u5143</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>\u5176\u4e2d\uff0c<span translate=no>_^_1_^_</span>\u662f\u6cbf\u5e8f\u5217\u7ef4\u5ea6\u7684\u7ebf\u6027\u53d8\u6362\uff0c<span translate=no>_^_2_^_</span>\u662f\u9010\u5143\u7d20\u4e58\u6cd5\u3002<span translate=no>_^_3_^_</span>\u88ab\u5206\u6210\u4e24\u4e2a\u5927\u5c0f\u76f8\u7b49\u7684\u90e8\u5206\uff0c<span translate=no>_^_4_^_</span>\u5e76<span translate=no>_^_5_^_</span>\u6cbf\u7740\u901a\u9053\u5c3a\u5bf8\uff08\u5d4c\u5165\u7ef4\u5ea6\uff09\u3002</p>\n",
+ "<h2>gMLP Block</h2>\n<p>Each block does the following transformations to input embeddings <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the sequence length and <span translate=no>_^_2_^_</span> is the dimensionality of the embeddings:</p>\n<span translate=no>_^_3_^_</span><p>where <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are learnable projection weights. <span translate=no>_^_6_^_</span> is the Spacial Gating Unit defined below. Output dimensionality of <span translate=no>_^_7_^_</span> will be half of <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span> is an activation function such as <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLU</a>.</p>\n": "<h2>gmLP Block</h2>\n<p>\u6bcf\u4e2a\u6a21\u5757\u5bf9\u8f93\u5165\u5d4c\u5165\u8fdb\u884c\u4ee5\u4e0b\u8f6c\u6362\uff0c<span translate=no>_^_0_^_</span>\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u5e8f\u5217\u957f\u5ea6\uff0c<span translate=no>_^_2_^_</span>\u662f\u5d4c\u5165\u7684\u7ef4\u5ea6\uff1a</p>\n<span translate=no>_^_3_^_</span><p>\u5176\u4e2d<span translate=no>_^_4_^_</span>\u548c<span translate=no>_^_5_^_</span>\u662f\u53ef\u5b66\u4e60\u7684\u6295\u5f71\u6743\u91cd\u3002<span translate=no>_^_6_^_</span>\u662f\u4e0b\u9762\u5b9a\u4e49\u7684\u7a7a\u95f4\u95e8\u63a7\u5355\u5143\u3002\u7684\u8f93\u51fa\u7ef4\u5ea6<span translate=no>_^_7_^_</span>\u5c06\u4e3a\u7684\u4e00\u534a<span translate=no>_^_8_^_</span>\u3002<span translate=no>_^_9_^_</span>\u662f\u4e00\u4e2a\u6fc0\u6d3b\u51fd\u6570\uff0c\u6bd4\u5982 <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLU</a>\u3002</p>\n",
+ "<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
+ "<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span>. The batch dimension should be of size <span translate=no>_^_2_^_</span> because this implementation supports only same mask for all samples in the batch. </p>\n": "<p><span translate=no>_^_0_^_</span>\u6709\u5f62\u72b6<span translate=no>_^_1_^_</span>\u3002\u6279\u6b21\u7ef4\u5ea6\u5e94\u4e3a size\uff0c<span translate=no>_^_2_^_</span>\u56e0\u4e3a\u6b64\u5b9e\u73b0\u4ec5\u652f\u6301\u6279\u6b21\u4e2d\u6240\u6709\u6837\u672c\u7684\u76f8\u540c\u63a9\u7801\u3002</p>\n",
+ "<p>Activation function <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6fc0\u6d3b\u529f\u80fd<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Add the shortcut connection </p>\n": "<p>\u6dfb\u52a0\u5feb\u6377\u65b9\u5f0f\u8fde\u63a5</p>\n",
+ "<p>Apply mask to the weights.</p>\n<p>If <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> then <span translate=no>_^_2_^_</span> will not get any information from token <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u5c06\u906e\u7f69\u5e94\u7528\u4e8e\u6743\u91cd\u3002</p>\n<p>\u5982\u679c<span translate=no>_^_0_^_</span>\u662f<span translate=no>_^_1_^_</span>\uff0c\u5219<span translate=no>_^_2_^_</span>\u4e0d\u4f1a\u4ece\u4ee4\u724c\u4e2d\u83b7\u53d6\u4efb\u4f55\u4fe1\u606f<span translate=no>_^_3_^_</span>\u3002</p>\n",
+ "<p>Check mask </p>\n": "<p>\u68c0\u67e5\u53e3\u7f69</p>\n",
+ "<p>Embedding size (required by <a href=\"../models.html#Encoder\">Encoder</a>. We use the encoder module from transformer architecture and plug <em>gMLP</em> block as a replacement for the <a href=\"../models.html#Encoder\">Transformer Layer</a>. </p>\n": "<p>\u5d4c\u5165\u5927\u5c0f\uff08\u7f16<a href=\"../models.html#Encoder\">\u7801\u5668</a>\u9700\u8981\u3002\u6211\u4eec\u4f7f\u7528\u53d8\u538b\u5668\u67b6\u6784\u4e2d\u7684\u7f16\u7801\u5668\u6a21\u5757\uff0c\u5e76\u63d2\u5165 <em>GmLP</em> \u6a21\u5757\u4f5c\u4e3a<a href=\"../models.html#Encoder\">\u53d8\u538b\u5668\u5c42</a>\u7684\u66ff\u4ee3\u54c1\u3002</p>\n",
+ "<p>Final projection <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u7ec8\u6295\u5f71<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Get sequence length </p>\n": "<p>\u83b7\u53d6\u5e8f\u5217\u957f\u5ea6</p>\n",
+ "<p>Get the weight matrix; truncate if larger than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u83b7\u53d6\u6743\u91cd\u77e9\u9635\uff1b\u5982\u679c\u5927\u4e8e<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Here we only support the same mask for all samples </p>\n": "<p>\u8fd9\u91cc\u6211\u4eec\u53ea\u652f\u6301\u6240\u6709\u6837\u672c\u4f7f\u7528\u76f8\u540c\u7684\u63a9\u7801</p>\n",
+ "<p>Keep a copy for shortcut connection </p>\n": "<p>\u4fdd\u7559\u4e00\u4efd\u7528\u4e8e\u5feb\u6377\u65b9\u5f0f\u8fde\u63a5\u7684\u526f\u672c</p>\n",
+ "<p>Normalization layer before applying <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e94\u7528\u4e4b\u524d\u7684\u6807\u51c6\u5316\u5c42<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Normalization layer fro Pre-Norm </p>\n": "<p>Pre-Norm \u7684\u6807\u51c6\u5316\u5c42</p>\n",
+ "<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u89c4\u8303\u5316<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Normalize <span translate=no>_^_0_^_</span> before <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u4e4b\u524d\u8fdb\u884c\u6807\u51c6\u5316<span translate=no>_^_1_^_</span></p>\n",
+ "<p>Projection and activation <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6295\u5c04\u548c\u6fc0\u6d3b<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Projection layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6295\u5f71\u5c42<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Remove the batch dimension </p>\n": "<p>\u79fb\u9664\u6279\u91cf\u7ef4\u5ea6</p>\n",
+ "<p>Spacial Gating Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7a7a\u95f4\u95e8\u63a7\u5355\u5143<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Split <span translate=no>_^_0_^_</span> into <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> </p>\n": "<p>\u62c6<span translate=no>_^_0_^_</span>\u5206\u4e3a<span translate=no>_^_1_^_</span>\u548c<span translate=no>_^_2_^_</span></p>\n",
+ "<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span>.</p>\n<p>The paper notes that it&#x27;s important to initialize weights to small values and the bias to <span translate=no>_^_2_^_</span>, so that during the initial training <span translate=no>_^_3_^_</span> is close to identity (apart from the split). </p>\n": "<p>\u91cd\u91cf<span translate=no>_^_0_^_</span>\u5728<span translate=no>_^_1_^_</span>\u3002</p>\n<p>\u672c\u6587\u6307\u51fa\uff0c\u91cd\u8981\u7684\u662f\u5c06\u6743\u91cd\u521d\u59cb\u5316\u4e3a\u8f83\u5c0f\u7684\u503c\uff0c\u5e76\u5c06\u504f\u5dee\u521d\u59cb\u5316<span translate=no>_^_3_^_</span>\u4e3a<span translate=no>_^_2_^_</span>\uff0c\u8fd9\u6837\u5728\u521d\u59cb\u8bad\u7ec3\u671f\u95f4\u5c31\u63a5\u8fd1\u8eab\u4efd\uff08\u62c6\u5206\u9664\u5916\uff09\u3002</p>\n",
+ "<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span></p>\n<p>The paper notes that it&#x27;s important to initialize bias to <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u91cd\u91cf<span translate=no>_^_0_^_</span>\u5728<span translate=no>_^_1_^_</span></p>\n<p>\u672c\u6587\u6307\u51fa\uff0c\u5c06\u504f\u89c1\u521d\u59cb\u5316\u4e3a<span translate=no>_^_2_^_</span>.</p>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the dimensionality (<span translate=no>_^_1_^_</span>) of <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the dimensionality of <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the length of the token sequence (<span translate=no>_^_6_^_</span>)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u7684\u7ef4\u5ea6 (<span translate=no>_^_1_^_</span>)<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f\u7684\u7ef4\u5ea6<span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u662f\u4ee4\u724c\u5e8f\u5217\u7684\u957f\u5ea6 (<span translate=no>_^_6_^_</span>)</li></ul>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the dimensionality of <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the sequence length</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u7684\u7ef4\u5ea6<span translate=no>_^_1_^_</span></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u5e8f\u5217\u957f\u5ea6</li></ul>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the input <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens  among each other. The last dimension of size <span translate=no>_^_5_^_</span> is the batch, which we have in other transformer  implementations and was left for compatibility.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6<span translate=no>_^_1_^_</span>\u7684\u8f93\u5165<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>is \u662f\u5f62\u72b6\u7684\u5e03\u5c14\u63a9\u7801<span translate=no>_^_4_^_</span>\uff0c\u7528\u4e8e\u63a7\u5236\u6807\u8bb0\u5728\u5f7c\u6b64\u4e4b\u95f4\u7684\u53ef\u89c1\u6027\u3002\u5c3a\u5bf8\u7684\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6<span translate=no>_^_5_^_</span>\u662f\u6279\u6b21\uff0c\u8fd9\u662f\u6211\u4eec\u5728\u5176\u4ed6\u53d8\u538b\u5668\u5b9e\u73b0\u4e2d\u4f7f\u7528\u7684\uff0c\u4e3a\u4e86\u517c\u5bb9\u6027\u800c\u7559\u4e0b\u3002</li></ul>\n",
+ "<ul><li><span translate=no>_^_0_^_</span> is the input embedding tensor <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens  among each other.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u7684\u8f93\u5165\u5d4c\u5165<span translate=no>_^_1_^_</span>\u5f20\u91cf<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f\u5f62\u72b6\u7684\u5e03\u5c14\u63a9\u7801<span translate=no>_^_4_^_</span>\uff0c\u7528\u4e8e\u63a7\u5236\u6807\u8bb0\u5728\u5f7c\u6b64\u4e4b\u95f4\u7684\u53ef\u89c1\u6027\u3002</li></ul>\n",
+ "Pay Attention to MLPs (gMLP)": "\u6ce8\u610f MLP (gMLP)",
+ "This is an annotated implementation/tutorial of Pay Attention to MLPs (gMLP) in PyTorch.": "\u8fd9\u662f PyTorch \u4e2d\u6ce8\u610f MLP\uff08GmLP\uff09\u7684\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0/\u6559\u7a0b\u3002"
+}
@@ -0,0 +1,32 @@
+{
+ "<h1><a href=\"index.html\">Pay Attention to MLPs (gMLP)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">gMLP model</a>. The paper also applies a Stochastic Depth regularization where some layers are removed randomly during training. We have not implemented that here.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h1><a href=\"index.html\">MLP (gMLP) \u306e\u5b9f\u9a13\u306b\u3054\u6ce8\u76ee\u304f\u3060\u3055\u3044</a></h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f gMLP \u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30a2\u30ce\u30c6\u30fc\u30b7\u30e7\u30f3\u4ed8\u304d PyTorch \u5b9f\u9a13\u3067\u3059\u3002</a>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\u4e00\u90e8\u306e\u5c64\u304c\u30e9\u30f3\u30c0\u30e0\u306b\u524a\u9664\u3055\u308c\u308b\u78ba\u7387\u7684\u6df1\u5ea6\u6b63\u5247\u5316\u3082\u9069\u7528\u3057\u3066\u3044\u307e\u3059\u3002\u3053\u3053\u3067\u306f\u5b9f\u88c5\u3057\u3066\u3044\u307e\u305b\u3093\u3002</p>\n<p>\u3053\u308c\u306f\u3001<a href=\"../basic/autoregressive_experiment.html\">\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u5df1\u56de\u5e30NLP\u30bf\u30b9\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u69cb\u6210\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
+ "<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u3001<a href=\"../basic/autoregressive_transformer.html\">\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u5df1\u56de\u5e30NLP\u30bf\u30b9\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u69cb\u6210\u304b\u3089\u7d99\u627f\u3055\u308c\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
+ "<h3>Create a gMLP block</h3>\n": "<h3>gMLP \u30d6\u30ed\u30c3\u30af\u3092\u4f5c\u6210\u3059\u308b</h3>\n",
+ "<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
+ "<p> </p>\n": "<p></p>\n",
+ "<p><span translate=no>_^_0_^_</span> for gMLP projection layer </p>\n": "<p><span translate=no>_^_0_^_</span>gMLP \u30d7\u30ed\u30b8\u30a7\u30af\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u7528</p>\n",
+ "<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
+ "<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
+ "<p>Model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba</p>\n",
+ "<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
+ "<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
+ "<p>Replace the encoder layer with a gMLP layer </p>\n": "<p>\u30a8\u30f3\u30b3\u30fc\u30c0\u5c64\u3092gMLP\u5c64\u306b\u7f6e\u304d\u63db\u3048\u308b</p>\n",
+ "<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
+ "<p>Set model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
+ "<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
+ "<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
+ "<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
+ "<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
+ "<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
+ "<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
+ "<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
+ "<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
+ "<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
+ "<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
+ "<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
+ "<p>gMLP Block </p>\n": "<p>GmLP \u30d6\u30ed\u30c3\u30af</p>\n",
+ "Pay Attention to MLPs (gMLP) Experiment": "MLP (gMLP) \u306e\u5b9f\u9a13\u306b\u3054\u6ce8\u76ee\u304f\u3060\u3055\u3044",
+ "This experiment trains a gMLP based model on Tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u57fa\u306b gMLP \u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
+}
@@ -0,0 +1,32 @@
+{
+ "<h1><a href=\"index.html\">Pay Attention to MLPs (gMLP)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">gMLP model</a>. The paper also applies a Stochastic Depth regularization where some layers are removed randomly during training. We have not implemented that here.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">MLPs (GMLP) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4</a> \u0d9a\u0dbb\u0db1\u0dca\u0db1</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">GMLP \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0dbb\u0dca\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0dc3\u0db8\u0dc4\u0dbb \u0dc3\u0dca\u0dae\u0dbb \u0d85\u0dc4\u0db9\u0dd4 \u0dbd\u0dd9\u0dc3 \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0db6\u0db1 Stochastic Depth regularization \u0daf \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d85\u0daf\u0dcf\u0dc5 \u0dc0\u0dda. \u0d85\u0db4\u0dd2 \u0d91\u0dba \u0db8\u0dd9\u0dc4\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb \u0db1\u0dd0\u0dad. </p>\n<p>\u0db8\u0dd9\u0dba\u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda <a href=\"../basic/autoregressive_experiment.html\">\u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba-\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0db8\u0dad \u0dba. </p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_0_^_</span></a></p>\n",
+ "<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba <a href=\"../basic/autoregressive_transformer.html\">\u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba-\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9a\u0dbb\u0dca\u0dad\u0dc0\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0dda. </p>\n",
+ "<h3>Create a gMLP block</h3>\n": "<h3>GMLP\u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca \u0d91\u0d9a\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h3>\n",
+ "<h3>Transformer configurations</h3>\n": "<h3>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
+ "<p> </p>\n": "<p> </p>\n",
+ "<p><span translate=no>_^_0_^_</span> for gMLP projection layer </p>\n": "<p><span translate=no>_^_0_^_</span> GMLP \u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0dab \u0dc3\u0dca\u0dae\u0dbb\u0dba \u0dc3\u0db3\u0dc4\u0dcf </p>\n",
+ "<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Model size </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
+ "<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
+ "<p>Replace the encoder layer with a gMLP layer </p>\n": "<p>\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca\u0dc3\u0dca\u0dad\u0dbb\u0dba GMLP \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
+ "<p>Set model size </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
+ "<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
+ "<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
+ "<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
+ "<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
+ "<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
+ "<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
+ "<p>gMLP Block </p>\n": "<p>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3\u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca </p>\n",
+ "Pay Attention to MLPs (gMLP) Experiment": "MLPs (GMLP) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1",
+ "This experiment trains a gMLP based model on Tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db8\u0dad GMLP \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
+}
@@ -0,0 +1,32 @@
+{
+ "<h1><a href=\"index.html\">Pay Attention to MLPs (gMLP)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">gMLP model</a>. The paper also applies a Stochastic Depth regularization where some layers are removed randomly during training. We have not implemented that here.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h1><a href=\"index.html\">\u6ce8\u610f mLP (gmLP)</a> \u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3 <a href=\"index.html\">gmLP \u6a21\u578b</a>\u3002\u672c\u6587\u8fd8\u5e94\u7528\u4e86\u968f\u673a\u6df1\u5ea6\u6b63\u5219\u5316\uff0c\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u4f1a\u968f\u673a\u5220\u9664\u4e00\u4e9b\u56fe\u5c42\u3002\u6211\u4eec\u6ca1\u6709\u5728\u8fd9\u91cc\u5b9e\u73b0\u8fd9\u4e00\u70b9\u3002</p>\n<p>\u8fd9\u662f\u57fa\u4e8e<a href=\"../basic/autoregressive_experiment.html\">\u7b80\u5355\u53d8\u6362\u5668\u81ea\u56de\u5f52 NLP \u4efb\u52a1\u7684\u8bad\u7ec3\u5faa\u73af\u548c\u914d\u7f6e</a>\u3002</p>\n",
+ "<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea<a href=\"../basic/autoregressive_transformer.html\">\u8bad\u7ec3\u5faa\u73af\u548c\u7b80\u5355\u53d8\u538b\u5668\u81ea\u56de\u5f52\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4efb\u52a1\u7684\u914d\u7f6e</a>\u3002</p>\n",
+ "<h3>Create a gMLP block</h3>\n": "<h3>\u521b\u5efa\u4e00\u4e2a GmLP \u533a\u5757</h3>\n",
+ "<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
+ "<p> </p>\n": "<p></p>\n",
+ "<p><span translate=no>_^_0_^_</span> for gMLP projection layer </p>\n": "<p><span translate=no>_^_0_^_</span>\u7528\u4e8e gMLP \u6295\u5f71\u5c42</p>\n",
+ "<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
+ "<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
+ "<p>Model size </p>\n": "<p>\u578b\u53f7\u5c3a\u5bf8</p>\n",
+ "<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
+ "<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
+ "<p>Replace the encoder layer with a gMLP layer </p>\n": "<p>\u5c06\u7f16\u7801\u5668\u5c42\u66ff\u6362\u4e3a GmLP \u5c42</p>\n",
+ "<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
+ "<p>Set model size </p>\n": "<p>\u8bbe\u7f6e\u6a21\u578b\u5c3a\u5bf8</p>\n",
+ "<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
+ "<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
+ "<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
+ "<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
+ "<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
+ "<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
+ "<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
+ "<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
+ "<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
+ "<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
+ "<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
+ "<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
+ "<p>gMLP Block </p>\n": "<p>gmLP Block</p>\n",
+ "Pay Attention to MLPs (gMLP) Experiment": "\u6ce8\u610f MLP (gMLP) \u5b9e\u9a8c",
+ "This experiment trains a gMLP based model on Tiny Shakespeare dataset.": "\u672c\u5b9e\u9a8c\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u57fa\u4e8e GmLP \u7684\u6a21\u578b\u3002"
+}
@@ -0,0 +1,4 @@
+{
+ "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">the training code</a> for a gMLP model based autoregressive model. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044</a></h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.08050\">MLP\u306b\u6ce8\u610f\u3057\u3066</a>\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a>\u3002</p>\n<p><strong>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30b2\u30fc\u30c6\u30a3\u30f3\u30b0\u3092\u5099\u3048\u305f\u591a\u5c64\u30d1\u30fc\u30bb\u30d7\u30c8\u30ed\u30f3\uff08MLP\uff09\u30d9\u30fc\u30b9\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\uff08GMLP\u3068\u540d\u4ed8\u3051\u3089\u308c\u3066\u3044\u307e\u3059\uff09\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002</strong><span translate=no>_^_0_^_</span><em>gMLP</em> \u30d6\u30ed\u30c3\u30af\u306e\u30b9\u30bf\u30c3\u30af\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059</p>\u3002\n<p><a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">gMLP\u30e2\u30c7\u30eb\u30d9\u30fc\u30b9\u306e\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059</a>\u3002</p>\n",
+ "Pay Attention to MLPs (gMLP)": "MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044"
+}
@@ -0,0 +1,4 @@
+{
+ "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">the training code</a> for a gMLP model based autoregressive model.</p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_1_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a></h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/2105.08050\">MLPs \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a> . </p>\n<p>\u0db8\u0dd9\u0db8\u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0db6\u0dc4\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0db4\u0dbb\u0dca\u0dc3\u0dd9\u0db4\u0dca\u0da7\u0dca\u0dbb\u0ddd\u0db1\u0dca (\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3) \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0d9a\u0dca \u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0d9f \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0d85\u0dad\u0dbb \u0d92\u0dc0\u0dcf <strong>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3</strong>\u0dbd\u0dd9\u0dc3 \u0db1\u0db8\u0dca \u0d9a\u0dbb\u0dba\u0dd2. \u0d91\u0dba <span translate=no>_^_0_^_</span> <em>GMLP</em> \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dad\u0ddc\u0d9c\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0db1\u0dca\u0dc0\u0dd2\u0dad \u0dc0\u0dda. </p>\n<p>GMLP\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf <a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dda\u0dad\u0dba</a> \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_1_^_</span></a> </p>\n",
+ "Pay Attention to MLPs (gMLP)": "MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1"
+}
@@ -0,0 +1,4 @@
+{
+ "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">the training code</a> for a gMLP model based autoregressive model. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">\u6ce8\u610f MLP (GmLP)</a></h1>\n<p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u5bf9\u300a<a href=\"https://arxiv.org/abs/2105.08050\">\u6ce8\u610f MLP\u300b\u4e00\u6587\u7684</a>\u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u4ecb\u7ecd\u4e86\u4e00\u79cd\u57fa\u4e8e\u591a\u5c42\u611f\u77e5\u5668\uff08MLP\uff09\u7684\u5e26\u6709\u95e8\u63a7\u7684\u67b6\u6784\uff0c\u4ed6\u4eec\u5c06\u5176\u547d\u540d\u4e3a <strong>gmLP</strong>\u3002\u5b83\u7531\u4e00\u5806<span translate=no>_^_0_^_</span> <em>gmLP</em> \u5757\u7ec4\u6210\u3002</p>\n<p>\u8fd9\u662f\u57fa<a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">\u4e8e GmLP \u6a21\u578b\u7684\u81ea\u56de\u5f52\u6a21\u578b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
+ "Pay Attention to MLPs (gMLP)": "\u6ce8\u610f MLP (gMLP)"
+}