chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"<h1>FNet: Mixing Tokens with Fourier Transforms</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"../mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a <span translate=no>_^_0_^_</span> more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>.</p>\n<h2>Mixing tokens with two Fourier transforms</h2>\n<p>We apply Fourier transform along the hidden dimension (embedding dimension) and then along the sequence dimension.</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>where <span translate=no>_^_2_^_</span> is the embedding input, <span translate=no>_^_3_^_</span> stands for the fourier transform and <span translate=no>_^_4_^_</span> stands for the real component in complex numbers.</p>\n<p>This is very simple to implement on PyTorch - just 1 line of code. The paper suggests using a precomputed DFT matrix and doing matrix multiplication to get the Fourier transformation.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for using a FNet based model for classifying <a href=\"https://paperswithcode.com/dataset/ag-news\">AG News</a>.</p>\n": "<h1>FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408</h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.03824\">FNet: \u30c8\u30fc\u30af\u30f3\u3092\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3068\u6df7\u5408\u3059\u308b\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n<p><em>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<a href=\"../mha.html\"><a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u5c64\u30922\u3064\u306e\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u7f6e\u304d\u63db\u3048\u3066\u30c8\u30fc\u30af\u30f3\u3092\u6df7\u5408\u3057\u307e\u3059</a></a>\u3002</em><span translate=no>_^_0_^_</span>\u3053\u308c\u306f\u81ea\u5df1\u6ce8\u610f\u3088\u308a\u3082\u52b9\u7387\u7684\u3067\u3059\u3002<a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE</a> \u30d9\u30f3\u30c1\u30de\u30fc\u30af\u3067\u306f\u3001\u81ea\u5df1\u6ce8\u610f\u3088\u308a\u3082\u3053\u308c\u3092\u4f7f\u7528\u3057\u305f\u5834\u5408\u306e\u7cbe\u5ea6\u306e\u4f4e\u4e0b\u306f\u7d04 92%</p> \u3067\u3059\u3002\n<h2>2 \u3064\u306e\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408</h2>\n<p>\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u975e\u8868\u793a\u6b21\u5143 (\u57cb\u3081\u8fbc\u307f\u6b21\u5143) \u306b\u6cbf\u3063\u3066\u9069\u7528\u3057\u3001\u6b21\u306b\u30b7\u30fc\u30b1\u30f3\u30b9\u6b21\u5143\u306b\u6cbf\u3063\u3066\u9069\u7528\u3057\u307e\u3059\u3002</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>\u3053\u3053\u3067\u3001<span translate=no>_^_2_^_</span>\u306f\u57cb\u3081\u8fbc\u307f\u5165\u529b\u3067\u3001<span translate=no>_^_3_^_</span><span translate=no>_^_4_^_</span>\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u8868\u3057\u3001\u8907\u7d20\u6570\u306e\u5b9f\u6570\u6210\u5206\u3092\u8868\u3057\u307e\u3059\u3002</p>\n<p>\u3053\u308c\u3092PyTorch\u306b\u5b9f\u88c5\u3059\u308b\u306e\u306f\u3068\u3066\u3082\u7c21\u5358\u3067\u3059\u3002\u305f\u3063\u305f1\u884c\u306e\u30b3\u30fc\u30c9\u3067\u3059\u3002\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u4e8b\u524d\u306b\u8a08\u7b97\u3055\u308c\u305fDFT\u884c\u5217\u3092\u4f7f\u7528\u3057\u3001\u884c\u5217\u306e\u4e57\u7b97\u3092\u884c\u3063\u3066\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u884c\u3046\u3053\u3068\u3092\u63d0\u6848\u3057\u3066\u3044\u307e\u3059</p>\u3002\n<p>\u4ee5\u4e0b\u306f<a href=\"experiment.html\">\u3001<a href=\"https://paperswithcode.com/dataset/ag-news\">FNet\u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u3092\u4f7f\u7528\u3057\u3066AG</a> News\u3092\u5206\u985e\u3059\u308b\u305f\u3081\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3067\u3059</a>\u3002</p>\n",
|
||||
"<h2>FNet - Mix tokens</h2>\n<p>This module simply implements <span translate=no>_^_0_^_</span></p>\n<p>The structure of this module is made similar to a <a href=\"../mha.html\">standard attention module</a> so that we can simply replace it.</p>\n": "<h2>FNet-\u30df\u30c3\u30af\u30b9\u30c8\u30fc\u30af\u30f3</h2>\n<p>\u3053\u306e\u30e2\u30b8\u30e5\u30fc\u30eb\u306f\u5358\u7d14\u306b\u5b9f\u88c5\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n<p>\u3053\u306e\u30e2\u30b8\u30e5\u30fc\u30eb\u306e\u69cb\u9020\u306f\u3001<a href=\"../mha.html\">\u6a19\u6e96\u7684\u306a\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30b8\u30e5\u30fc\u30eb\u3068\u540c\u69d8\u306e\u69cb\u9020\u306b\u306a\u3063\u3066\u3044\u308b\u305f\u3081</a>\u3001\u7c21\u5358\u306b\u4ea4\u63db\u3067\u304d\u307e\u3059\u3002</p>\n",
|
||||
"<p> The <a href=\"../mha.html\">normal attention module</a> can be fed with different token embeddings for <span translate=no>_^_0_^_</span>,<span translate=no>_^_1_^_</span>, and <span translate=no>_^_2_^_</span> and a mask.</p>\n<p>We follow the same function signature so that we can replace it directly.</p>\n<p>For FNet mixing, <span translate=no>_^_3_^_</span> and masking is not possible. Shape of <span translate=no>_^_4_^_</span> (and <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>) is <span translate=no>_^_7_^_</span>.</p>\n": "<p><a href=\"../mha.html\">\u901a\u5e38\u306e\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30b8\u30e5\u30fc\u30eb\u306b\u306f</a>\u3001\u3001\u3001<span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u30de\u30b9\u30af\u306b\u3055\u307e\u3056\u307e\u306a\u30c8\u30fc\u30af\u30f3\u3092\u57cb\u3081\u8fbc\u3080\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n<p>\u540c\u3058\u95a2\u6570\u30b7\u30b0\u30cd\u30c1\u30e3\u306b\u5f93\u3046\u306e\u3067\u3001\u76f4\u63a5\u7f6e\u63db\u3067\u304d\u307e\u3059\u3002</p>\n<p>FNet\u30df\u30ad\u30b7\u30f3\u30b0\u306e\u5834\u5408<span translate=no>_^_3_^_</span>\u3001\u30de\u30b9\u30ad\u30f3\u30b0\u306f\u3067\u304d\u307e\u305b\u3093\u3002<span translate=no>_^_4_^_</span>(<span translate=no>_^_5_^_</span>\u3068<span translate=no>_^_6_^_</span>) \u306e\u5f62\u306f\u3067\u3059<span translate=no>_^_7_^_</span>\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>,<span translate=no>_^_1_^_</span>, and <span translate=no>_^_2_^_</span> all should be equal to <span translate=no>_^_3_^_</span> for token mixing </p>\n": "<p><span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u3001<span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u305d\u3057\u3066\u30c8\u30fc\u30af\u30f3\u306e\u30df\u30ad\u30b7\u30f3\u30b0\u3067\u306f\u3059\u3079\u3066\u304c\u7b49\u3057\u304f\u306a\u3051\u308c\u3070\u306a\u308a\u307e\u305b\u3093</p>\n",
|
||||
"<p>Apply the Fourier transform along the hidden (embedding) dimension <span translate=no>_^_0_^_</span></p>\n<p>The output of the Fourier transform is a tensor of <a href=\"https://pytorch.org/docs/stable/complex_numbers.html\">complex numbers</a>. </p>\n": "<p>\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u975e\u8868\u793a (\u57cb\u3081\u8fbc\u307f) \u6b21\u5143\u306b\u6cbf\u3063\u3066\u9069\u7528\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n<p><a href=\"https://pytorch.org/docs/stable/complex_numbers.html\">\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306e\u51fa\u529b\u306f\u8907\u7d20\u6570\u306e\u30c6\u30f3\u30bd\u30eb\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>Apply the Fourier transform along the sequence dimension <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u6b21\u5143\u306b\u6cbf\u3063\u3066\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u9069\u7528\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Assign to <span translate=no>_^_0_^_</span> for clarity </p>\n": "<p><span translate=no>_^_0_^_</span>\u308f\u304b\u308a\u3084\u3059\u3044\u3088\u3046\u306b\u5272\u308a\u5f53\u3066\u308b</p>\n",
|
||||
"<p>Get the real component <span translate=no>_^_0_^_</span> </p>\n": "<p>\u672c\u7269\u306e\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8\u3092\u624b\u306b\u5165\u308c\u3088\u3046 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Token mixing doesn't support masking. i.e. all tokens will see all other token embeddings. </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u30df\u30ad\u30b7\u30f3\u30b0\u306f\u30de\u30b9\u30ad\u30f3\u30b0\u3092\u30b5\u30dd\u30fc\u30c8\u3057\u3066\u3044\u307e\u305b\u3093\u3002\u3064\u307e\u308a\u3001\u3059\u3079\u3066\u306e\u30c8\u30fc\u30af\u30f3\u306b\u4ed6\u306e\u3059\u3079\u3066\u306e\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u304c\u8868\u793a\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408",
|
||||
"This is an annotated implementation/tutorial of FNet in PyTorch.": "\u3053\u308c\u306f PyTorch \u306b\u304a\u3051\u308b FNet \u306e\u30a2\u30ce\u30c6\u30fc\u30b7\u30e7\u30f3\u4ed8\u304d\u306e\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"<h1>FNet: Mixing Tokens with Fourier Transforms</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"../mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a <span translate=no>_^_0_^_</span> more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>.</p>\n<h2>Mixing tokens with two Fourier transforms</h2>\n<p>We apply Fourier transform along the hidden dimension (embedding dimension) and then along the sequence dimension.</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>where <span translate=no>_^_2_^_</span> is the embedding input, <span translate=no>_^_3_^_</span> stands for the fourier transform and <span translate=no>_^_4_^_</span> stands for the real component in complex numbers.</p>\n<p>This is very simple to implement on PyTorch - just 1 line of code. The paper suggests using a precomputed DFT matrix and doing matrix multiplication to get the Fourier transformation.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for using a FNet based model for classifying <a href=\"https://paperswithcode.com/dataset/ag-news\">AG News</a>.</p>\n": "<h1>FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408</h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/2105.03824\">FNet\uff1a\u5c06\u4ee3\u5e01\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408\u300b\u7684 PyTor</a> <a href=\"https://pytorch.org\">ch</a> \u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u7528\u4e24\u4e2a<a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u5085\u91cc\u53f6\u53d8</a>\u6362\u53d6\u4ee3\u4e86<a href=\"../mha.html\">\u81ea\u6211\u6ce8\u610f\u529b\u5c42</a>\uff0c\u4ee5<em>\u6df7\u5408</em>\u4ee4\u724c\u3002\u8fd9\u6bd4\u81ea\u6211\u6ce8\u610f\u529b<span translate=no>_^_0_^_</span>\u66f4\u6709\u6548\u3002\u5728 GLUE <a href=\"https://paperswithcode.com/dataset/glue\">\u57fa\u51c6\u6d4b\u8bd5</a>\u4e2d\uff0c<a href=\"https://paperswithcode.com/method/bert\">BERT</a> \u4f7f\u7528\u5b83\u800c\u4e0d\u662f\u81ea\u6211\u6ce8\u610f\u529b\u7684\u51c6\u786e\u6027\u635f\u5931\u7ea6\u4e3a92\uff05\u3002</p>\n<h2>\u5c06\u4ee4\u724c\u4e0e\u4e24\u4e2a\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408</h2>\n<p>\u6211\u4eec\u6cbf\u9690\u85cf\u7ef4\u5ea6\uff08\u5d4c\u5165\u7ef4\u5ea6\uff09\u5e94\u7528\u5085\u91cc\u53f6\u53d8\u6362\uff0c\u7136\u540e\u6cbf\u5e8f\u5217\u7ef4\u5ea6\u5e94\u7528\u5085\u91cc\u53f6\u53d8\u6362\u3002</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>\u5176\u4e2d<span translate=no>_^_2_^_</span>\u662f\u5d4c\u5165\u8f93\u5165\uff0c<span translate=no>_^_3_^_</span>\u4ee3\u8868\u5085\u91cc\u53f6\u53d8\u6362\uff0c<span translate=no>_^_4_^_</span>\u4ee3\u8868\u590d\u6570\u4e2d\u7684\u5b9e\u5206\u91cf\u3002</p>\n<p>\u8fd9\u5728 PyTorch \u4e0a\u5b9e\u73b0\u8d77\u6765\u975e\u5e38\u7b80\u5355-\u53ea\u9700\u4e00\u884c\u4ee3\u7801\u3002\u672c\u6587\u5efa\u8bae\u4f7f\u7528\u9884\u5148\u8ba1\u7b97\u7684DFT\u77e9\u9635\u5e76\u8fdb\u884c\u77e9\u9635\u4e58\u6cd5\u6765\u83b7\u5f97\u5085\u91cc\u53f6\u53d8\u6362\u3002</p>\n<p><a href=\"experiment.html\">\u4ee5\u4e0b\u662f\u4f7f\u7528\u57fa\u4e8e FNet \u7684\u6a21\u578b\u5bf9 <a href=\"https://paperswithcode.com/dataset/ag-news\">AG News</a> \u8fdb\u884c\u5206\u7c7b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
|
||||
"<h2>FNet - Mix tokens</h2>\n<p>This module simply implements <span translate=no>_^_0_^_</span></p>\n<p>The structure of this module is made similar to a <a href=\"../mha.html\">standard attention module</a> so that we can simply replace it.</p>\n": "<h2>FNet-\u6df7\u5408\u4ee3\u5e01</h2>\n<p>\u8fd9\u4e2a\u6a21\u5757\u7b80\u5355\u5730\u5b9e\u73b0\u4e86<span translate=no>_^_0_^_</span></p>\n<p>\u8be5\u6a21\u5757\u7684\u7ed3\u6784\u7c7b\u4f3c\u4e8e<a href=\"../mha.html\">\u6807\u51c6\u7684\u6ce8\u610f\u529b\u6a21\u5757</a>\uff0c\u56e0\u6b64\u6211\u4eec\u53ef\u4ee5\u7b80\u5355\u5730\u66ff\u6362\u5b83\u3002</p>\n",
|
||||
"<p> The <a href=\"../mha.html\">normal attention module</a> can be fed with different token embeddings for <span translate=no>_^_0_^_</span>,<span translate=no>_^_1_^_</span>, and <span translate=no>_^_2_^_</span> and a mask.</p>\n<p>We follow the same function signature so that we can replace it directly.</p>\n<p>For FNet mixing, <span translate=no>_^_3_^_</span> and masking is not possible. Shape of <span translate=no>_^_4_^_</span> (and <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>) is <span translate=no>_^_7_^_</span>.</p>\n": "<p><a href=\"../mha.html\">\u666e\u901a\u6ce8\u610f\u529b\u6a21\u5757</a>\u53ef\u4ee5\u4f7f\u7528<span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u548c\u7684\u4e0d\u540c\u4ee4\u724c\u5d4c\u5165<span translate=no>_^_2_^_</span>\u4ee5\u53ca\u63a9\u7801\u8fdb\u884c\u9988\u9001\u3002</p>\n<p>\u6211\u4eec\u9075\u5faa\u76f8\u540c\u7684\u51fd\u6570\u7b7e\u540d\uff0c\u4ee5\u4fbf\u6211\u4eec\u53ef\u4ee5\u76f4\u63a5\u66ff\u6362\u5b83\u3002</p>\n<p>\u5bf9\u4e8e FNet \u6df7\u5408<span translate=no>_^_3_^_</span>\uff0c\u5c4f\u853d\u662f\u4e0d\u53ef\u80fd\u7684\u3002<span translate=no>_^_4_^_</span>\uff08\u548c<span translate=no>_^_5_^_</span>\u548c<span translate=no>_^_6_^_</span>\uff09\u7684\u5f62\u72b6\u4e3a<span translate=no>_^_7_^_</span>\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>,<span translate=no>_^_1_^_</span>, and <span translate=no>_^_2_^_</span> all should be equal to <span translate=no>_^_3_^_</span> for token mixing </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3001\uff0c<span translate=no>_^_3_^_</span>\u5bf9\u4e8e\u4ee4\u724c\u6df7\u5408\uff0cal<span translate=no>_^_2_^_</span> l \u5e94\u8be5\u7b49\u4e8e</p>\n",
|
||||
"<p>Apply the Fourier transform along the hidden (embedding) dimension <span translate=no>_^_0_^_</span></p>\n<p>The output of the Fourier transform is a tensor of <a href=\"https://pytorch.org/docs/stable/complex_numbers.html\">complex numbers</a>. </p>\n": "<p>\u6cbf\u9690\u85cf\uff08\u5d4c\u5165\uff09\u7ef4\u5ea6\u5e94\u7528\u5085\u91cc\u53f6\u53d8\u6362<span translate=no>_^_0_^_</span></p>\n<p>\u5085\u91cc\u53f6\u53d8\u6362\u7684\u8f93\u51fa\u662f<a href=\"https://pytorch.org/docs/stable/complex_numbers.html\">\u590d\u6570</a>\u5f20\u91cf\u3002</p>\n",
|
||||
"<p>Apply the Fourier transform along the sequence dimension <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6cbf\u5e8f\u5217\u7ef4\u5ea6\u5e94\u7528\u5085\u91cc\u53f6\u53d8\u6362<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Assign to <span translate=no>_^_0_^_</span> for clarity </p>\n": "<p>\u4e3a\u4e86\u6e05\u695a\u8d77\u89c1\uff0c<span translate=no>_^_0_^_</span>\u8bf7\u5206\u914d\u7ed9</p>\n",
|
||||
"<p>Get the real component <span translate=no>_^_0_^_</span> </p>\n": "<p>\u83b7\u53d6\u771f\u6b63\u7684\u7ec4\u4ef6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Token mixing doesn't support masking. i.e. all tokens will see all other token embeddings. </p>\n": "<p>\u4ee4\u724c\u6df7\u5408\u4e0d\u652f\u6301\u63a9\u7801\u3002\u5373\u6240\u6709\u4ee4\u724c\u90fd\u5c06\u770b\u5230\u6240\u6709\u5176\u4ed6\u4ee4\u724c\u5d4c\u5165\u3002</p>\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408",
|
||||
"This is an annotated implementation/tutorial of FNet in PyTorch.": "\u8fd9\u662f PyTorch \u4e2d FNet \u7684\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0/\u6559\u7a0b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">FNet</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">FNet model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_classification.html\">general training loop and configurations for AG News classification task</a>.</p>\n": "<h1><a href=\"index.html\">FNet \u5b9f\u9a13</a></h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f\u3001FNet\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u306ePyTorch\u5b9f\u9a13\u3067\u3059\u3002</a></p>\n<p>\u3053\u308c\u306f\u3001<a href=\"../../experiments/nlp_classification.html\">AG News\u5206\u985e\u30bf\u30b9\u30af\u306e\u4e00\u822c\u7684\u306a\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u69cb\u6210\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h1>Transformer based classifier model</h1>\n": "<h1>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30d9\u30fc\u30b9\u306e\u5206\u985e\u5668\u30e2\u30c7\u30eb</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u4ee5\u4e0b\u304b\u3089\u7d99\u627f\u3055\u308c\u307e\u3059 <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create <span translate=no>_^_0_^_</span> module that can replace the self-attention in <a href=\"../models.html#TransformerLayer\">transformer encoder layer</a> .</p>\n": "<p><span translate=no>_^_0_^_</span><a href=\"../models.html#TransformerLayer\">\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u5c64\u306e\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u306b\u4ee3\u308f\u308b\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u4f5c\u6210\u3057\u3066\u304f\u3060\u3055\u3044</a>\u3002</p>\n",
|
||||
"<p> Create classification model</p>\n": "<p>\u5206\u985e\u30e2\u30c7\u30eb\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Classification model </p>\n": "<p>\u5206\u985e\u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Get logits for classification.</p>\n<p>We set the <span translate=no>_^_0_^_</span> token at the last position of the sequence. This is extracted by <span translate=no>_^_1_^_</span>, where <span translate=no>_^_2_^_</span> is of shape <span translate=no>_^_3_^_</span> </p>\n": "<p>\u5206\u985e\u7528\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97\u3057\u307e\u3059\u3002</p>\n<p><span translate=no>_^_0_^_</span>\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u6700\u5f8c\u306e\u4f4d\u7f6e\u306b\u30c8\u30fc\u30af\u30f3\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u5f62\u72b6\u304c\u3069\u3053\u306b\u3042\u308b\u304b\u306b\u3088\u3063\u3066\u62bd\u51fa\u3055\u308c\u307e\u3059 <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3057\u307e\u3059\uff08\u30c8\u30ec\u30fc\u30ca\u30fc\u306fRNN\u3067\u3082\u4f7f\u7528\u3055\u308c\u308b\u305f\u3081\u30012\u756a\u76ee\u306e\u5024\u306f\u72b6\u614b\u7528\u3067\u3059\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u5909\u5727\u5668\u69cb\u6210 (\u30c7\u30d5\u30a9\u30eb\u30c8\u3068\u540c\u3058)</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use <a href=\"index.html\">FNet</a> instead of self-a ttention </p>\n": "<p><a href=\"index.html\">\u81ea\u5df1\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u306e\u4ee3\u308f\u308a\u306bFNet\u3092\u4f7f\u3046</a></p>\n",
|
||||
"<p>Use world level tokenizer </p>\n": "<p>\u30ef\u30fc\u30eb\u30c9\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../models.html#Encoder\">\u5909\u5727\u5668\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span><a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u306f\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3059 (\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u4ed8\u304d)</a></li>\n</ul><li><span translate=no>_^_2_^_</span><a href=\"../models.html#Generator\">\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u6700\u5f8c\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u3067\u3059</a>\u3002</li>\n",
|
||||
"FNet Experiment": "FNet \u5b9f\u9a13",
|
||||
"This experiment trains a FNet based model on AG News dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001AG News\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u57fa\u3065\u3044\u3066FNet\u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">FNet</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">FNet model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_classification.html\">general training loop and configurations for AG News classification task</a>.</p>\n": "<h1><a href=\"index.html\">FNet</a> \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">FNet \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf PyTorch \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p>\u0db8\u0dd9\u0dba\u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda <a href=\"../../experiments/nlp_classification.html\">AG \u0db4\u0dca\u0dbb\u0dc0\u0dd8\u0dad\u0dca\u0dad\u0dd2 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0db8\u0dad \u0dba. </p>\n",
|
||||
"<h1>Transformer based classifier model</h1>\n": "<h1>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0db1\u0dca\u0db1\u0dda <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create <span translate=no>_^_0_^_</span> module that can replace the self-attention in <a href=\"../models.html#TransformerLayer\">transformer encoder layer</a> .</p>\n": "<p> <a href=\"../models.html#TransformerLayer\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda</a> \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 <span translate=no>_^_0_^_</span> \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p> Create classification model</p>\n": "<p> \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p>Classification model </p>\n": "<p>\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get logits for classification.</p>\n<p>We set the <span translate=no>_^_0_^_</span> token at the last position of the sequence. This is extracted by <span translate=no>_^_1_^_</span>, where <span translate=no>_^_2_^_</span> is of shape <span translate=no>_^_3_^_</span> </p>\n": "<p>\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1. </p>\n<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dba\u0dda \u0d85\u0db4\u0dd2 <span translate=no>_^_0_^_</span> \u0da7\u0ddd\u0d9a\u0db1\u0dba \u0dc3\u0d9a\u0dc3\u0dca \u0d9a\u0dbb\u0db8\u0dd4. \u0db8\u0dd9\u0dba \u0d8b\u0db4\u0dd4\u0da7\u0dcf \u0d9c\u0db1\u0dd4 \u0dbd\u0db6\u0db1\u0dca\u0db1\u0dda <span translate=no>_^_1_^_</span>\u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca <span translate=no>_^_2_^_</span> \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf </p>? <span translate=no>_^_3_^_</span>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dda\u0dad\u0db1 \u0d9a\u0dca\u0dbb\u0db8 \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0dcf\u0db7\u0db4\u0dca\u0dbb\u0dad\u0dd2 results \u0dbd (\u0daf\u0dd9\u0dc0\u0db1 \u0d85\u0d9c\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dda, \u0db8\u0db1\u0dca\u0daf \u0d85\u0db4\u0d9c\u0dda \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4 RNs \u0dc3\u0db8\u0d9f \u0daf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2) </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca (\u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dbd\u0dd9\u0dc3) </p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use <a href=\"index.html\">FNet</a> instead of self-a ttention </p>\n": "<p>\u0dc3\u0dca\u0dc0\u0dba\u0d82-\u0d87\u0dbd\u0dc0\u0dd3\u0db8\u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7 <a href=\"index.html\">FNet</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use world level tokenizer </p>\n": "<p>\u0dbd\u0ddd\u0d9a\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"../models.html#Encoder\">\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a> </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0da7\u0ddd\u0d9a\u0db1\u0dca <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba (\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab \u0dc3\u0db8\u0d9f)</a> </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1 <a href=\"../models.html#Generator\">\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dba\u0dd2</a> . </li></ul>\n",
|
||||
"FNet Experiment": "FNet \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains a FNet based model on AG News dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 AG News \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db8\u0dad FNet \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">FNet</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">FNet model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_classification.html\">general training loop and configurations for AG News classification task</a>.</p>\n": "<h1><a href=\"index.html\">FNet</a> \u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u4e2a\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3 <a href=\"index.html\">FNet \u6a21\u578b</a>\u3002</p>\n<p>\u8fd9\u662f\u57fa\u4e8e <a href=\"../../experiments/nlp_classification.html\">AG News \u5206\u7c7b\u4efb\u52a1\u7684\u4e00\u822c\u8bad\u7ec3\u5faa\u73af\u548c\u914d\u7f6e</a>\u3002</p>\n",
|
||||
"<h1>Transformer based classifier model</h1>\n": "<h1>\u57fa\u4e8e\u53d8\u538b\u5668\u7684\u5206\u7c7b\u5668\u6a21\u578b</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create <span translate=no>_^_0_^_</span> module that can replace the self-attention in <a href=\"../models.html#TransformerLayer\">transformer encoder layer</a> .</p>\n": "<p>\u521b\u5efa\u53ef\u4ee5\u53d6\u4ee3<a href=\"../models.html#TransformerLayer\">\u53d8\u538b\u5668\u7f16\u7801\u5668\u5c42</a>\u4e2d\u7684\u81ea\u6211\u6ce8\u610f\u529b\u7684<span translate=no>_^_0_^_</span>\u6a21\u5757\u3002</p>\n",
|
||||
"<p> Create classification model</p>\n": "<p>\u521b\u5efa\u5206\u7c7b\u6a21\u578b</p>\n",
|
||||
"<p>Classification model </p>\n": "<p>\u5206\u7c7b\u6a21\u578b</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Get logits for classification.</p>\n<p>We set the <span translate=no>_^_0_^_</span> token at the last position of the sequence. This is extracted by <span translate=no>_^_1_^_</span>, where <span translate=no>_^_2_^_</span> is of shape <span translate=no>_^_3_^_</span> </p>\n": "<p>\u83b7\u53d6\u5206\u7c7b\u65e5\u5fd7\u3002</p>\n<p>\u6211\u4eec\u5c06\u4ee4<span translate=no>_^_0_^_</span>\u724c\u8bbe\u7f6e\u5728\u5e8f\u5217\u7684\u6700\u540e\u4e00\u4e2a\u4f4d\u7f6e\u3002\u8fd9\u662f\u7531\uff0cwher<span translate=no>_^_1_^_</span> e<span translate=no>_^_2_^_</span> \u662f\u5f62\u72b6\u63d0\u53d6</p>\u7684<span translate=no>_^_3_^_</span>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f7f\u7528\u4f4d\u7f6e\u7f16\u7801\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c\uff08\u7b2c\u4e8c\u4e2a\u503c\u7528\u4e8e\u72b6\u6001\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u8bad\u7ec3\u5668\u4e5f\u4e0e RNN \u4e00\u8d77\u4f7f\u7528\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u53d8\u538b\u5668\u914d\u7f6e\uff08\u4e0e\u9ed8\u8ba4\u503c\u76f8\u540c\uff09</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use <a href=\"index.html\">FNet</a> instead of self-a ttention </p>\n": "<p>\u4f7f\u7528 <a href=\"index.html\">FNet</a> \u800c\u4e0d\u662f\u81ea\u6211\u5173\u6ce8</p>\n",
|
||||
"<p>Use world level tokenizer </p>\n": "<p>\u4f7f\u7528\u4e16\u754c\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668<a href=\"../models.html#Encoder\">\u7f16\u7801\u5668</a></li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c<a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u5d4c\u5165\u6a21\u5757\uff08\u5e26\u6709\u4f4d\u7f6e\u7f16\u7801\uff09</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u7ed9<a href=\"../models.html#Generator\">\u51fa logit \u7684\u6700\u540e\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42</a>\u3002</li></ul>\n",
|
||||
"FNet Experiment": "FNet \u5b9e\u9a8c",
|
||||
"This experiment trains a FNet based model on AG News dataset.": "\u8be5\u5b9e\u9a8c\u57fa\u4e8eAG News\u6570\u636e\u96c6\u8bad\u7ec3\u4e00\u4e2a\u57fa\u4e8eFNet\u7684\u6a21\u578b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers//mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408</a></h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.03824\">FNet: \u30c8\u30fc\u30af\u30f3\u3092\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3068\u6df7\u5408\u3059\u308b\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n<p><em>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<a href=\"https://nn.labml.ai/transformers//mha.html\"><a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u81ea\u5df1\u6ce8\u610f\u5c64\u30922\u3064\u306e\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u7f6e\u304d\u63db\u3048\u3066\u30c8\u30fc\u30af\u30f3\u3092\u6df7\u5408\u3057\u307e\u3059</a></a>\u3002</em>\u3053\u308c\u306f\u81ea\u5df1\u51e6\u7406\u3088\u308a\u30827\u500d\u52b9\u7387\u7684\u3067\u3059\u3002<a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE</a> \u30d9\u30f3\u30c1\u30de\u30fc\u30af\u3067\u306f\u3001\u81ea\u5df1\u6ce8\u610f\u3088\u308a\u3082\u3053\u308c\u3092\u4f7f\u7528\u3057\u305f\u5834\u5408\u306e\u7cbe\u5ea6\u306e\u4f4e\u4e0b\u306f\u7d04 92%</p> \u3067\u3059\u3002\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers//mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: \u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0db4\u0dbb\u0dd2\u0dab\u0dcf\u0db8\u0db1\u0dba \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1 \u0db8\u0dd2\u0dc1\u0dca\u0dbb \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a></h1>\n<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://arxiv.org/abs/2105.03824\">FNet \u0dc4\u0dd2 <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8: \u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dca \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1 \u0db8\u0dd2\u0dc1\u0dca\u0dbb</a> \u0d9a\u0dd2\u0dbb\u0dd3\u0db8. </p>\n<p>\u0db8\u0dd9\u0db8\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://nn.labml.ai/transformers//mha.html\">\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dad\u0dbb\u0dba</a> \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2 <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dd9\u0d9a\u0d9a\u0dca \u0da7\u0ddd\u0d9a\u0db1 <em>\u0db8\u0dd2\u0dc1\u0dca\u0dbb</em> \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba</a> \u0d9a\u0dbb\u0dba\u0dd2. \u0db8\u0dd9\u0dba 7X \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0da7 \u0dc0\u0da9\u0dcf \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8 \u0dc0\u0dda. \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dd9\u0db1\u0dca \u0db8\u0dd9\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db1\u0dd0\u0dad\u0dd2\u0dc0\u0dd3\u0db8 GLUE <a href=\"https://paperswithcode.com/dataset/glue\">\u0db8\u0dd2\u0dab\u0dd4\u0db8\u0dca \u0daf\u0dab\u0dca\u0da9\u0dda</a> <a href=\"https://paperswithcode.com/method/bert\">BERT</a> \u0dc3\u0db3\u0dc4\u0dcf 92% \u0d9a\u0dca \u0db4\u0db8\u0dab \u0dc0\u0dda. </p>\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet: \u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0db4\u0dbb\u0dd2\u0dab\u0dcf\u0db8\u0db1\u0dba \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1 \u0db8\u0dd2\u0dc1\u0dca\u0dbb \u0d9a\u0dd2\u0dbb\u0dd3\u0db8"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers//mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408</a></h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/2105.03824\">FNet\uff1a\u5c06\u4ee3\u5e01\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408\u300b\u7684 PyTor</a> <a href=\"https://pytorch.org\">ch</a> \u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u7528\u4e24\u4e2a<a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u5085\u91cc\u53f6\u53d8</a>\u6362\u53d6\u4ee3\u4e86<a href=\"https://nn.labml.ai/transformers//mha.html\">\u81ea\u6211\u6ce8\u610f\u529b\u5c42</a>\uff0c\u4ee5<em>\u6df7\u5408</em>\u4ee4\u724c\u3002\u8fd9\u6bd4\u81ea\u6211\u6ce8\u610f\u529b\u9ad87\u500d\u3002\u5728 GLUE <a href=\"https://paperswithcode.com/dataset/glue\">\u57fa\u51c6\u6d4b\u8bd5</a>\u4e2d\uff0c<a href=\"https://paperswithcode.com/method/bert\">BERT</a> \u4f7f\u7528\u5b83\u800c\u4e0d\u662f\u81ea\u6211\u6ce8\u610f\u529b\u7684\u51c6\u786e\u6027\u635f\u5931\u7ea6\u4e3a92\uff05\u3002</p>\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408"
|
||||
}
|
||||
Reference in New Issue
Block a user