chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"<h1>Transformers</h1>\n": "<h1>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc</h1>\n",
|
||||
"<h2><a href=\"aft/index.html\">Attention Free Transformer</a></h2>\n": "<h2><a href=\"aft/index.html\">\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d5\u30ea\u30fc\u5909\u5727\u5668</a></h2>\n",
|
||||
"<h2><a href=\"alibi/index.html\">Attention with Linear Biases</a></h2>\n": "<h2><a href=\"alibi/index.html\">\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f</a></h2>\n",
|
||||
"<h2><a href=\"compressive/index.html\">Compressive Transformer</a></h2>\n": "<h2><a href=\"compressive/index.html\">\u5727\u7e2e\u5909\u5727\u5668</a></h2>\n",
|
||||
"<h2><a href=\"fast_weights/index.html\">Fast Weights Transformer</a></h2>\n": "<h2><a href=\"fast_weights/index.html\">\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30c8\u30e9\u30f3\u30b9</a></h2>\n",
|
||||
"<h2><a href=\"feedback/index.html\">Feedback Transformer</a></h2>\n": "<h2><a href=\"feedback/index.html\">\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u5909\u5727\u5668</a></h2>\n",
|
||||
"<h2><a href=\"fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h2>\n": "<h2><a href=\"fnet/index.html\">FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408</a></h2>\n",
|
||||
"<h2><a href=\"glu_variants/simple.html\">GLU Variants</a></h2>\n": "<h2><a href=\"glu_variants/simple.html\">GLU \u30d0\u30ea\u30a2\u30f3\u30c8</a></h2>\n",
|
||||
"<h2><a href=\"gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h2>\n": "<h2><a href=\"gmlp/index.html\">MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044</a></h2>\n",
|
||||
"<h2><a href=\"gpt/index.html\">GPT Architecture</a></h2>\n": "<h2><a href=\"gpt/index.html\">GPT \u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3</a></h2>\n",
|
||||
"<h2><a href=\"hour_glass/index.html\">Hourglass</a></h2>\n": "<h2><a href=\"hour_glass/index.html\">\u7802\u6642\u8a08</a></h2>\n",
|
||||
"<h2><a href=\"knn/index.html\">kNN-LM</a></h2>\n": "<h2><a href=\"knn/index.html\">KNN-LM</a></h2>\n",
|
||||
"<h2><a href=\"mlm/index.html\">Masked Language Model</a></h2>\n": "<h2><a href=\"mlm/index.html\">\u30de\u30b9\u30af\u8a00\u8a9e\u30e2\u30c7\u30eb</a></h2>\n",
|
||||
"<h2><a href=\"mlp_mixer/index.html\">MLP-Mixer: An all-MLP Architecture for Vision</a></h2>\n": "<h2><a href=\"mlp_mixer/index.html\">MLP\u30df\u30ad\u30b5\u30fc:\u30d3\u30b8\u30e7\u30f3\u7528\u306e\u30aa\u30fc\u30ebMLP\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3</a></h2>\n",
|
||||
"<h2><a href=\"primer_ez/index.html\">Primer EZ</a></h2>\n": "<h2><a href=\"primer_ez/index.html\">\u30d7\u30e9\u30a4\u30de\u30fc EZ</a></h2>\n",
|
||||
"<h2><a href=\"retro/index.html\">RETRO</a></h2>\n": "<h2><a href=\"retro/index.html\">\u30ec\u30c8\u30ed</a></h2>\n",
|
||||
"<h2><a href=\"rope/index.html\">Rotary Positional Embeddings</a></h2>\n": "<h2><a href=\"rope/index.html\">\u30ed\u30fc\u30bf\u30ea\u30fc\u30dd\u30b8\u30b7\u30e7\u30ca\u30eb\u30a8\u30f3\u30d9\u30c7\u30a3\u30f3\u30b0</a></h2>\n",
|
||||
"<h2><a href=\"switch/index.html\">Switch Transformer</a></h2>\n": "<h2><a href=\"switch/index.html\">\u30b9\u30a4\u30c3\u30c1\u30c8\u30e9\u30f3\u30b9</a></h2>\n",
|
||||
"<h2><a href=\"vit/index.html\">Vision Transformer (ViT)</a></h2>\n": "<h2><a href=\"vit/index.html\">\u30d3\u30b8\u30e7\u30f3\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc (ViT)</a></h2>\n",
|
||||
"<h2><a href=\"xl/index.html\">Transformer XL</a></h2>\n": "<h2><a href=\"xl/index.html\">\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc XL</a></h2>\n",
|
||||
"<p>This implements Attention with Linear Biases (ALiBi).</p>\n": "<p>\u3053\u308c\u306f\u3001\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\uff08AliBi\uff09\u306b\u3088\u308b\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u3092\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059\u3002</p>\n",
|
||||
"<p>This implements Rotary Positional Embeddings (RoPE)</p>\n": "<p>\u3053\u308c\u306f\u30ed\u30fc\u30bf\u30ea\u30fc\u30fb\u30dd\u30b8\u30b7\u30e7\u30ca\u30eb\u30fb\u30a8\u30f3\u30d9\u30c7\u30a3\u30f3\u30b0 (RoPE) \u3092\u5b9f\u88c5\u3057\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<p>This implements Transformer XL model using <a href=\"xl/relative_mha.html\">relative multi-head attention</a></p>\n": "<p>\u3053\u308c\u306f\u3001<a href=\"xl/relative_mha.html\">\u76f8\u5bfe\u7684\u306a\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u3092\u4f7f\u7528\u3057\u305f\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fcXL\u30e2\u30c7\u30eb\u3092\u5b9f\u88c5\u3057\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<p>This implements the Retrieval-Enhanced Transformer (RETRO).</p>\n": "<p>\u3053\u308c\u306f\u691c\u7d22\u5f37\u5316\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc (RETRO) \u3092\u5b9f\u88c5\u3057\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<p>This is a miniature implementation of the paper <a href=\"https://arxiv.org/abs/2101.03961\">Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity</a>. Our implementation only has a few million parameters and doesn't do model parallel distributed training. It does single GPU training but we implement the concept of switching as described in the paper.</p>\n": "<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u306e\u300c<a href=\"https://arxiv.org/abs/2101.03961\">\u30b9\u30a4\u30c3\u30c1\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\uff1a\u30b7\u30f3\u30d7\u30eb\u3067\u52b9\u7387\u7684\u306a\u30b9\u30d1\u30fc\u30b9\u6027\u3092\u5099\u3048\u305f1\u5146\u30d1\u30e9\u30e1\u30fc\u30bf\u30e2\u30c7\u30eb\u3078\u306e\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0</a>\u300d\u306e\u30df\u30cb\u30c1\u30e5\u30a2\u5b9f\u88c5\u3067\u3059\u3002\u79c1\u305f\u3061\u306e\u5b9f\u88c5\u306b\u306f\u6570\u767e\u4e07\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3057\u304b\u306a\u304f\u3001\u30e2\u30c7\u30eb\u306e\u4e26\u5217\u5206\u6563\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306f\u884c\u3044\u307e\u305b\u3093\u3002\u30b7\u30f3\u30b0\u30ebGPU\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u884c\u3044\u307e\u3059\u304c\u3001\u8ad6\u6587\u3067\u8aac\u660e\u3055\u308c\u3066\u3044\u308b\u3088\u3046\u306b\u30b9\u30a4\u30c3\u30c1\u30f3\u30b0\u306e\u6982\u5ff5\u3092\u5b9f\u88c5\u3057\u3066\u3044\u307e\u3059</p>\u3002\n",
|
||||
"<p>This is an implementation of GPT-2 architecture.</p>\n": "<p>\u3053\u308c\u306f GPT-2 \u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is an implementation of Masked Language Model used for pre-training in paper <a href=\"https://arxiv.org/abs/1810.04805\">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/1810.04805\">BERT\uff1a\u8a00\u8a9e\u7406\u89e3\u306e\u305f\u3081\u306e\u30c7\u30a3\u30fc\u30d7\u53cc\u65b9\u5411\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306e\u4e8b\u524d\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u300d\u306e\u4e8b\u524d\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306b\u4f7f\u7528\u3055\u308c\u305f\u30de\u30b9\u30af\u8a00\u8a9e\u30e2\u30c7\u30eb\u306e\u5b9f\u88c5\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>This is an implementation of compressive transformer that extends upon <a href=\"xl/index.html\">Transformer XL</a> by compressing the oldest memories to give a longer attention span.</p>\n": "<p>\u3053\u308c\u306f\u5727\u7e2e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306e\u5b9f\u88c5\u3067\u3001<a href=\"xl/index.html\">Transformer XL\u3092\u62e1\u5f35\u3057\u305f\u3082\u306e\u3067</a>\u3001\u6700\u3082\u53e4\u3044\u30e1\u30e2\u30ea\u3092\u5727\u7e2e\u3057\u3066\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30d1\u30f3\u3092\u9577\u304f\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/1911.00172\">Generalization through Memorization: Nearest Neighbor Language Models</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/1911.00172\">\u8a18\u61b6\u306b\u3088\u308b\u4e00\u822c\u5316\uff1a\u6700\u8fd1\u508d\u8a00\u8a9e\u30e2\u30c7\u30eb</a>\u300d\u3068\u3044\u3046\u8ad6\u6587\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2002.05202\">GLU Variants Improve Transformer</a>.</p>\n": "<p>\u3053\u308c\u306f\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2002.05202\">GLU\u30d0\u30ea\u30a2\u30f3\u30c8\u6539\u826f\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc</a>\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2002.09402\">Accessing Higher-level Representations in Sequential Transformers with Feedback Memory</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/2002.09402\">\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30e1\u30e2\u30ea\u3092\u7528\u3044\u305f\u30b7\u30fc\u30b1\u30f3\u30b7\u30e3\u30eb\u30fb\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306e\u9ad8\u4f4d\u8868\u73fe\u3078\u306e\u30a2\u30af\u30bb\u30b9\u300d\u3068\u3044\u3046\u8ad6\u6587\u306e\u5b9f\u88c5\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2010.11929\">An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/2010.11929\">\u753b\u50cf\u306f16x16\u306e\u8a00\u8449\u306b\u5024\u3059\u308b\u300d\u3068\u3044\u3046\u8ad6\u6587\u300c\u5927\u898f\u6a21\u753b\u50cf\u8a8d\u8b58\u306e\u305f\u3081\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002</a></p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2102.11174\">Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001PyTorch\u306e\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2102.11174\">\u30ea\u30cb\u30a2\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u5bc6\u304b\u306b\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30e1\u30e2\u30ea\u30b7\u30b9\u30c6\u30e0\u300d\u306e\u5b9f\u88c5\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.01601\">MLP-Mixer: An all-MLP Architecture for Vision</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.01601\">MLP\u30df\u30ad\u30b5\u30fc\uff1a\u30d3\u30b8\u30e7\u30f3\u7528\u306e\u30aa\u30fc\u30ebMLP\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3</a>\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n": "<p>\u3053\u308c\u306f\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.03824\">FNet: \u30c8\u30fc\u30af\u30f3\u3092\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3068\u6df7\u5408\u3059\u308b</a>\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.08050\">MLP\u306b\u6ce8\u610f\u3092\u6255\u3046</a>\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.14103\">An Attention Free Transformer</a>.</p>\n": "<p>\u3053\u308c\u306f\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.14103\">\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30d5\u30ea\u30fc\u30fb\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc</a>\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2109.08668\">Primer: Searching for Efficient Transformers for Language Modeling</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/2109.08668\">\u5165\u9580\u66f8\uff1a\u8a00\u8a9e\u30e2\u30c7\u30ea\u30f3\u30b0\u306e\u305f\u3081\u306e\u52b9\u7387\u7684\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306e\u63a2\u6c42\u300d\u3068\u3044\u3046\u8ad6\u6587\u306e\u5b9f\u88c5\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2110.13711\">Hierarchical Transformers Are More Efficient Language Models</a></p>\n": "<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/2110.13711\">\u968e\u5c64\u578b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u3088\u308a\u52b9\u7387\u7684\u306a\u8a00\u8a9e\u30e2\u30c7\u30eb</a>\u300d\u3068\u3044\u3046\u8ad6\u6587\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n",
|
||||
"<p>This module contains <a href=\"https://pytorch.org/\">PyTorch</a> implementations and explanations of original transformer from paper <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a>, and derivatives and enhancements of it.</p>\n": "</a><p>\u3053\u306e\u30e2\u30b8\u30e5\u30fc\u30eb\u306b\u306f\u3001<a href=\"https://pytorch.org/\">PyTorch\u306e\u5b9f\u88c5\u3068\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/1706.03762\">Attention IsAll You Need</a>\u300d\u306b\u63b2\u8f09\u3055\u308c\u305f\u30aa\u30ea\u30b8\u30ca\u30eb\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306e\u8aac\u660e\u3001\u304a\u3088\u3073\u305d\u306e\u6d3e\u751f\u54c1\u3068\u62e1\u5f35\u6a5f\u80fd\u304c\u542b\u307e\u308c\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<ul><li><a href=\"mha.html\">Multi-head attention</a> </li>\n<li><a href=\"models.html\">Transformer Encoder and Decoder Models</a> </li>\n<li><a href=\"feed_forward.html\">Position-wise Feed Forward Network (FFN)</a> </li>\n<li><a href=\"positional_encoding.html\">Fixed positional encoding</a></li></ul>\n": "<ul><li><a href=\"mha.html\">\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3</a></li>\n<li><a href=\"models.html\">\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u304a\u3088\u3073\u30c7\u30b3\u30fc\u30c0\u30e2\u30c7\u30eb</a></li>\n<li><a href=\"feed_forward.html\">\u4f4d\u7f6e\u5225\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af (FFN)</a></li>\n<li><a href=\"positional_encoding.html\">\u56fa\u5b9a\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0</a></li></ul>\n",
|
||||
"This is a collection of PyTorch implementations/tutorials of transformers and related techniques.": "\u3053\u308c\u306f\u3001\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3068\u95a2\u9023\u6280\u8853\u306e PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u306e\u30b3\u30ec\u30af\u30b7\u30e7\u30f3\u3067\u3059\u3002",
|
||||
"Transformers": "\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"<h1>Transformers</h1>\n": "<h1>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</h1>\n",
|
||||
"<h2><a href=\"aft/index.html\">Attention Free Transformer</a></h2>\n": "<h2><a href=\"aft/index.html\">\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db1\u0dd2\u0daf\u0dc4\u0dc3\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a></h2>\n",
|
||||
"<h2><a href=\"alibi/index.html\">Attention with Linear Biases</a></h2>\n": "<h2><a href=\"alibi/index.html\">\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0dc3\u0db8\u0d9f \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a></h2>\n",
|
||||
"<h2><a href=\"compressive/index.html\">Compressive Transformer</a></h2>\n": "<h2><a href=\"compressive/index.html\">\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a></h2>\n",
|
||||
"<h2><a href=\"fast_weights/index.html\">Fast Weights Transformer</a></h2>\n": "<h2><a href=\"fast_weights/index.html\">\u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a></h2>\n",
|
||||
"<h2><a href=\"feedback/index.html\">Feedback Transformer</a></h2>\n": "<h2><a href=\"feedback/index.html\">\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a></h2>\n",
|
||||
"<h2><a href=\"fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h2>\n": "<h2><a href=\"fnet/index.html\">FNet: \u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0db4\u0dbb\u0dd2\u0dab\u0dcf\u0db8\u0db1\u0dba \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1 \u0db8\u0dd2\u0dc1\u0dca\u0dbb \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a></h2>\n",
|
||||
"<h2><a href=\"glu_variants/simple.html\">GLU Variants</a></h2>\n": "<h2><a href=\"glu_variants/simple.html\">GLU \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf</a></h2>\n",
|
||||
"<h2><a href=\"gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h2>\n": "<h2><a href=\"gmlp/index.html\">MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a></h2>\n",
|
||||
"<h2><a href=\"gpt/index.html\">GPT Architecture</a></h2>\n": "<h2><a href=\"gpt/index.html\">Gpt \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba</a></h2>\n",
|
||||
"<h2><a href=\"hour_glass/index.html\">Hourglass</a></h2>\n": "<h2><a href=\"hour_glass/index.html\">Hourglass</a></h2>\n",
|
||||
"<h2><a href=\"knn/index.html\">kNN-LM</a></h2>\n": "<h2><a href=\"knn/index.html\">KN-\u0d91\u0dbd\u0dca \u0d91\u0db8\u0dca</a></h2>\n",
|
||||
"<h2><a href=\"mlm/index.html\">Masked Language Model</a></h2>\n": "<h2><a href=\"mlm/index.html\">\u0dc0\u0dd9\u0dc3\u0dca \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</a></h2>\n",
|
||||
"<h2><a href=\"mlp_mixer/index.html\">MLP-Mixer: An all-MLP Architecture for Vision</a></h2>\n": "<h2><a href=\"mlp_mixer/index.html\">\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3-\u0db8\u0dd2\u0d9a\u0dca\u0dc3\u0dbb\u0dca: \u0daf\u0dd0\u0d9a\u0dca\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dbb\u0dca\u0dc0 \u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0d9a\u0dca</a></h2>\n",
|
||||
"<h2><a href=\"primer_ez/index.html\">Primer EZ</a></h2>\n": "<h2><a href=\"primer_ez/index.html\">\u0db4\u0dca\u0dbb\u0dba\u0dd2\u0db8\u0dbb\u0dca EZ</a></h2>\n",
|
||||
"<h2><a href=\"retro/index.html\">RETRO</a></h2>\n": "<h2><a href=\"retro/index.html\">\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddc</a></h2>\n",
|
||||
"<h2><a href=\"rope/index.html\">Rotary Positional Embeddings</a></h2>\n": "<h2><a href=\"rope/index.html\">\u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca</a></h2>\n",
|
||||
"<h2><a href=\"switch/index.html\">Switch Transformer</a></h2>\n": "<h2><a href=\"switch/index.html\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dd2\u0da0\u0dba</a></h2>\n",
|
||||
"<h2><a href=\"vit/index.html\">Vision Transformer (ViT)</a></h2>\n": "<h2><a href=\"vit/index.html\">\u0daf\u0dbb\u0dca\u0dc1\u0db1 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca (VIT)</a></h2>\n",
|
||||
"<h2><a href=\"xl/index.html\">Transformer XL</a></h2>\n": "<h2><a href=\"xl/index.html\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca 40</a></h2>\n",
|
||||
"<p>This implements Attention with Linear Biases (ALiBi).</p>\n": "<p>\u0db8\u0dd9\u0dba\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d85\u0d9c\u0dad\u0dd3\u0db1\u0dca (\u0d85\u0dbd\u0dd2\u0db6\u0dd3) \u0dc3\u0db8\u0d9f \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>This implements Rotary Positional Embeddings (RoPE)</p>\n": "<p>\u0db8\u0dd9\u0dba\u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0dba\u0dd2</p>\n",
|
||||
"<p>This implements Transformer XL model using <a href=\"xl/relative_mha.html\">relative multi-head attention</a></p>\n": "<p>\u0db8\u0dd9\u0dba <a href=\"xl/relative_mha.html\">\u0dc3\u0dcf\u0db4\u0dda\u0d9a\u0dca\u0dc2 \u0db6\u0dc4\u0dd4-\u0dc4\u0dd2\u0dc3 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba</a>\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd2\u0db1\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d91\u0d9a\u0dca\u0dc3\u0dca\u0d91\u0dbd\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0dba\u0dd2</p>\n",
|
||||
"<p>This implements the Retrieval-Enhanced Transformer (RETRO).</p>\n": "<p>\u0db8\u0dd9\u0dba\u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2 \u0dc0\u0dd0\u0da9\u0dd2\u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dc5 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba (RETRO) \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>This is a miniature implementation of the paper <a href=\"https://arxiv.org/abs/2101.03961\">Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity</a>. Our implementation only has a few million parameters and doesn't do model parallel distributed training. It does single GPU training but we implement the concept of switching as described in the paper.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://arxiv.org/abs/2101.03961\">\u0dc3\u0dca\u0dc0\u0dd2\u0da0\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dd4\u0da9\u0dcf \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2: \u0dc3\u0dbb\u0dbd \u0dc4\u0dcf \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8 \u0dc3\u0dca\u0db4\u0dcf\u0dbb\u0dca\u0dc1\u0dd2\u0d9a\u0db8\u0d9a\u0dd2\u0db1\u0dca \u0dba\u0dd4\u0dad\u0dca \u0da7\u0dca\u0dbb\u0dd2\u0dbd\u0dd2\u0dba\u0db1 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba</a>\u0d9a\u0dd2\u0dbb\u0dd3\u0db8. \u0d85\u0db4\u0d9c\u0dda \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d87\u0dad\u0dca\u0dad\u0dda \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0db8\u0dd2\u0dbd\u0dd2\u0dba\u0db1 \u0d9a\u0dd2\u0dc4\u0dd2\u0db4\u0dba\u0d9a\u0dca \u0db4\u0db8\u0dab\u0d9a\u0dca \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0dc3\u0db8\u0dcf\u0db1\u0dca\u0dad\u0dbb\u0dc0 \u0db6\u0dd9\u0daf\u0dcf \u0dc4\u0dbb\u0dd2\u0db1 \u0dbd\u0daf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0d86\u0daf\u0dbb\u0dca\u0dc1\u0dba\u0da7 \u0db1\u0ddc\u0d9c\u0db1\u0dd3. \u0d91\u0dba \u0dad\u0db1\u0dd2 GPU \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dda \u0db1\u0db8\u0dd4\u0dad\u0dca \u0d85\u0db4\u0dd2 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dd2\u0dc3\u0dca\u0dad\u0dbb \u0d9a\u0dbb \u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0db8\u0dcf\u0dbb\u0dd4 \u0dc3\u0d82\u0d9a\u0dbd\u0dca\u0db4\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a. </p>\n",
|
||||
"<p>This is an implementation of GPT-2 architecture.</p>\n": "<p>\u0db8\u0dd9\u0dbaGPT-2 \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2. </p>\n",
|
||||
"<p>This is an implementation of Masked Language Model used for pre-training in paper <a href=\"https://arxiv.org/abs/1810.04805\">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dbd \u0db4\u0dd9\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db8\u0dcf\u0dc3\u0dca\u0da9\u0dca \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/1810.04805\">BERT: \u0db7\u0dcf\u0dc2\u0dcf \u0d85\u0dc0\u0db6\u0ddd\u0db0\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d9c\u0dd0\u0db9\u0dd4\u0dbb\u0dd4 \u0daf\u0dca\u0dc0\u0dd2\u0db4\u0dcf\u0dbb\u0dca\u0dc1\u0dca\u0dc0\u0dd2\u0d9a \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0d9a \u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0</a>. </p>\n",
|
||||
"<p>This is an implementation of compressive transformer that extends upon <a href=\"xl/index.html\">Transformer XL</a> by compressing the oldest memories to give a longer attention span.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"xl/index.html\">\u0d91\u0d9a\u0dca\u0dc3\u0dca\u0d91\u0dbd\u0dca \u0db8\u0dad \u0dc0\u0dd2\u0dc4\u0dd2\u0daf\u0dd9\u0db1 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dca \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dad\u0db8 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0daf\u0dd2\u0d9c\u0dd4 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0d9a\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dda. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/1911.00172\">Generalization through Memorization: Nearest Neighbor Language Models</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 <a href=\"https://arxiv.org/abs/1911.00172\">\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0da7\u0db4\u0dcf\u0da9\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8: \u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2 \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2</a>. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2002.05202\">GLU Variants Improve Transformer</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2 <a href=\"https://arxiv.org/abs/2002.05202\">GLU \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a>. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2002.09402\">Accessing Higher-level Representations in Sequential Transformers with Feedback Memory</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba <a href=\"https://arxiv.org/abs/2002.09402\">\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0db8\u0dad\u0d9a\u0dba \u0dc3\u0db8\u0d9f \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dbd \u0d89\u0dc4\u0dc5 \u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0db1\u0dd2\u0dbb\u0dd6\u0db4\u0dab\u0dba\u0db1\u0dca \u0dc0\u0dd9\u0dad \u0db4\u0dca\u0dbb\u0dc0\u0dda\u0dc1</a>\u0dc0\u0db1 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2010.11929\">An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale</a>.</p>\n": "<p>\u0db8\u0dd9\u0db8\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/2010.11929\">\u0dbb\u0dd6\u0db4\u0dba\u0d9a\u0dca \u0dc0\u0dbb\u0dca\u0dad\u0dca 16x16 \u0dc0\u0da0\u0db1: \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab \u0daf\u0dd3 \u0dbb\u0dd6\u0db4 \u0db4\u0dd2\u0dc5\u0dd2\u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a>. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2102.11174\">Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 <a href=\"https://arxiv.org/abs/2102.11174\">\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0dbb\u0dca\u0da0\u0dca \u0dc4\u0dd2 \u0dbb\u0dc4\u0dc3\u0dd2\u0db1\u0dca \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0db8\u0dad\u0d9a \u0db4\u0daf\u0dca\u0db0\u0dad\u0dd2</a>\u0dc0\u0dda. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.01601\">MLP-Mixer: An all-MLP Architecture for Vision</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://arxiv.org/abs/2105.01601\">\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3-\u0db8\u0dd2\u0d9a\u0dca\u0dc3\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2: \u0daf\u0dd0\u0d9a\u0dca\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dbb\u0dca\u0dc0 \u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba</a>. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2 <a href=\"https://arxiv.org/abs/2105.03824\">FNet: \u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0db4\u0ddd\u0dbb\u0dca\u0db8\u0dca \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1 \u0db8\u0dd2\u0dc1\u0dca\u0dbb</a>\u0d9a\u0dd2\u0dbb\u0dd3\u0db8. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 <a href=\"https://arxiv.org/abs/2105.08050\">MLPs \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a>. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.14103\">An Attention Free Transformer</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 <a href=\"https://arxiv.org/abs/2105.14103\">\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dbb\u0dc4\u0dd2\u0dad \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca</a>. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2109.08668\">Primer: Searching for Efficient Transformers for Language Modeling</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 <a href=\"https://arxiv.org/abs/2109.08668\">\u0db4\u0dca\u0dbb\u0dba\u0dd2\u0db8\u0dbb\u0dca: \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dd9\u0dc0\u0dd3\u0db8</a>. </p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2110.13711\">Hierarchical Transformers Are More Efficient Language Models</a></p>\n": "<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 <a href=\"https://arxiv.org/abs/2110.13711\">\u0db0\u0dd6\u0dbb\u0dcf\u0dc0\u0dbd\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc0\u0da9\u0dcf\u0dad\u0dca \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8 \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2</a></p>\n",
|
||||
"<p>This module contains <a href=\"https://pytorch.org/\">PyTorch</a> implementations and explanations of original transformer from paper <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a>, and derivatives and enhancements of it.</p>\n": "</a> <p>\u0db8\u0dd9\u0db8\u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba\u0dda <a href=\"https://pytorch.org/\">PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0dc3\u0dc4 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0db8\u0dd4\u0dbd\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0db4\u0dd0\u0dc4\u0dd0\u0daf\u0dd2\u0dbd\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0d85\u0da9\u0d82\u0d9c\u0dd4 <a href=\"https://arxiv.org/abs/1706.03762\">\u0dc0\u0dda \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0d94\u0db6\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc3\u0dd2\u0dba\u0dbd\u0dca\u0dbd</a> \u0dc3\u0dc4 \u0d91\u0dc4\u0dd2 \u0dc0\u0dca\u0dba\u0dd4\u0dad\u0dca\u0db4\u0db1\u0dca\u0db1\u0dba\u0db1\u0dca \u0dc3\u0dc4 \u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca. </p>\n",
|
||||
"<ul><li><a href=\"mha.html\">Multi-head attention</a> </li>\n<li><a href=\"models.html\">Transformer Encoder and Decoder Models</a> </li>\n<li><a href=\"feed_forward.html\">Position-wise Feed Forward Network (FFN)</a> </li>\n<li><a href=\"positional_encoding.html\">Fixed positional encoding</a></li></ul>\n": "<ul><li><a href=\"mha.html\">\u0db6\u0dc4\u0dd4-\u0dc4\u0dd2\u0dc3 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba</a> </li>\n<li><a href=\"models.html\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca \u0dc3\u0dc4 \u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a \u0d86\u0d9a\u0dd8\u0dad\u0dd2</a> </li>\n<li><a href=\"feed_forward.html\">\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0ddd\u0dc2\u0d9a \u0d89\u0daf\u0dd2\u0dbb\u0dd2 \u0da2\u0dcf\u0dbd\u0dba (FFN)</a> </li>\n<li><a href=\"positional_encoding.html\">\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab\u0dba</a></li></ul>\n",
|
||||
"This is a collection of PyTorch implementations/tutorials of transformers and related techniques.": "\u0db8\u0dd9\u0dba PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dc4 \u0d85\u0daf\u0dcf\u0dc5 \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dd3\u0dba \u0d9a\u0dca\u0dbb\u0db8 \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1 \u0d91\u0d9a\u0dad\u0dd4\u0dc0\u0d9a\u0dd2.",
|
||||
"Transformers": "\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca"
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"<h1>Transformers</h1>\n": "<h1>Transformers</h1>\n",
|
||||
"<h2><a href=\"aft/index.html\">Attention Free Transformer</a></h2>\n": "<h2><a href=\"aft/index.html\">\u65e0\u6ce8\u610f\u529b Transformer</a></h2>\n",
|
||||
"<h2><a href=\"alibi/index.html\">Attention with Linear Biases</a></h2>\n": "<h2><a href=\"alibi/index.html\">\u7ebf\u6027\u504f\u5dee\u6ce8\u610f\u529b</a></h2>\n",
|
||||
"<h2><a href=\"compressive/index.html\">Compressive Transformer</a></h2>\n": "<h2><a href=\"compressive/index.html\">\u538b\u7f29 Transformer</a></h2>\n",
|
||||
"<h2><a href=\"fast_weights/index.html\">Fast Weights Transformer</a></h2>\n": "<h2><a href=\"fast_weights/index.html\">\u5feb\u901f\u6743\u91cd Transformer</a></h2>\n",
|
||||
"<h2><a href=\"feedback/index.html\">Feedback Transformer</a></h2>\n": "<h2><a href=\"feedback/index.html\">\u81ea\u53cd\u9988 Transformer</a></h2>\n",
|
||||
"<h2><a href=\"fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h2>\n": "<h2><a href=\"fnet/index.html\">Fnet\uff1a\u4f7f\u7528\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408 token </a></h2>\n",
|
||||
"<h2><a href=\"glu_variants/simple.html\">GLU Variants</a></h2>\n": "<h2><a href=\"glu_variants/simple.html\">GLU \u53d8\u4f53</a></h2>\n",
|
||||
"<h2><a href=\"gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h2>\n": "<h2><a href=\"gmlp/index.html\">\u95e8\u63a7\u591a\u5c42\u611f\u77e5\u5668 (gMLP)</a></h2>\n",
|
||||
"<h2><a href=\"gpt/index.html\">GPT Architecture</a></h2>\n": "<h2><a href=\"gpt/index.html\">GPT \u67b6\u6784</a></h2>\n",
|
||||
"<h2><a href=\"hour_glass/index.html\">Hourglass</a></h2>\n": "<h2><a href=\"hour_glass/index.html\">\u6c99\u6f0f\u7f51\u7edc</a></h2>\n",
|
||||
"<h2><a href=\"knn/index.html\">kNN-LM</a></h2>\n": "<h2><a href=\"knn/index.html\">kNN-LM</a></h2>\n",
|
||||
"<h2><a href=\"mlm/index.html\">Masked Language Model</a></h2>\n": "<h2><a href=\"mlm/index.html\">\u63a9\u7801\u8bed\u8a00\u6a21\u578b</a></h2>\n",
|
||||
"<h2><a href=\"mlp_mixer/index.html\">MLP-Mixer: An all-MLP Architecture for Vision</a></h2>\n": "<h2><a href=\"mlp_mixer/index.html\">MLP-Mixer\uff1a\u4e00\u79cd\u7528\u4e8e\u89c6\u89c9\u7684\u5168 MLP \u67b6\u6784</a></h2>\n",
|
||||
"<h2><a href=\"primer_ez/index.html\">Primer EZ</a></h2>\n": "<h2><a href=\"primer_ez/index.html\">Primer</a></h2>\n",
|
||||
"<h2><a href=\"retro/index.html\">RETRO</a></h2>\n": "<h2><a href=\"retro/index.html\">RETRO</a></h2>\n",
|
||||
"<h2><a href=\"rope/index.html\">Rotary Positional Embeddings</a></h2>\n": "<h2><a href=\"rope/index.html\">\u65cb\u8f6c\u5f0f\u4f4d\u7f6e\u7f16\u7801</a></h2>\n",
|
||||
"<h2><a href=\"switch/index.html\">Switch Transformer</a></h2>\n": "<h2><a href=\"switch/index.html\">Switch Transformer</a></h2>\n",
|
||||
"<h2><a href=\"vit/index.html\">Vision Transformer (ViT)</a></h2>\n": "<h2><a href=\"vit/index.html\">\u89c6\u89c9 Transformer (ViT)</a></h2>\n",
|
||||
"<h2><a href=\"xl/index.html\">Transformer XL</a></h2>\n": "<h2><a href=\"xl/index.html\">Transformer XL</a></h2>\n",
|
||||
"<p>This implements Attention with Linear Biases (ALiBi).</p>\n": "<p>\u8fd9\u662f\u7ebf\u6027\u504f\u5dee\u6ce8\u610f\u529b\uff08 ALIBI \uff09\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This implements Rotary Positional Embeddings (RoPE)</p>\n": "<p>\u8fd9\u662f\u65cb\u8f6c\u5f0f\u4f4d\u7f6e\u7f16\u7801\uff08 ROPE \uff09\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This implements Transformer XL model using <a href=\"xl/relative_mha.html\">relative multi-head attention</a></p>\n": "<p>\u8fd9\u662f\u4f7f\u7528<a href=\"xl/relative_mha.html\">\u76f8\u5bf9\u591a\u5934\u6ce8\u610f\u529b</a>\u7684 Transformer XL \u6a21\u578b\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This implements the Retrieval-Enhanced Transformer (RETRO).</p>\n": "<p>\u8fd9\u662f\u5bf9\u68c0\u7d22\u589e\u5f3a Transformer \uff08 RETRO \uff09\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is a miniature implementation of the paper <a href=\"https://arxiv.org/abs/2101.03961\">Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity</a>. Our implementation only has a few million parameters and doesn't do model parallel distributed training. It does single GPU training but we implement the concept of switching as described in the paper.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/2101.03961\">\u300a Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity \u300b</a>\u7684\u4e00\u4e2a\u7b80\u5316\u5b9e\u73b0\u3002\u6211\u4eec\u7684\u5b9e\u73b0\u4ec5\u5305\u542b\u51e0\u767e\u4e07\u4e2a\u53c2\u6570\uff0c\u5e76\u4e14\u53ea\u5728\u5355 GPU \u4e0a\u8fdb\u884c\u8bad\u7ec3\uff0c\u4e0d\u6d89\u53ca\u5e76\u884c\u5206\u5e03\u5f0f\u8bad\u7ec3\uff0c\u4f46\u6211\u4eec\u4ecd\u7136\u5b9e\u73b0\u4e86\u8bba\u6587\u4e2d\u63cf\u8ff0\u7684 Switch \u6982\u5ff5\u3002</p>\n",
|
||||
"<p>This is an implementation of GPT-2 architecture.</p>\n": "<p>\u8fd9\u662f GPT-2 \u7ed3\u6784\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of Masked Language Model used for pre-training in paper <a href=\"https://arxiv.org/abs/1810.04805\">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/1810.04805\">\u300a BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding \u300b</a>\u4e2d\u7528\u4e8e\u9884\u8bad\u7ec3\u7684\u63a9\u7801\u8bed\u8a00\u6a21\u578b\u7684\u5b9e\u73b0</p>\n",
|
||||
"<p>This is an implementation of compressive transformer that extends upon <a href=\"xl/index.html\">Transformer XL</a> by compressing the oldest memories to give a longer attention span.</p>\n": "<p>\u8fd9\u662f\u4e00\u4e2a\u538b\u7f29transformer\u7684\u5b9e\u73b0\uff0c\u5b83\u5728<a href=\"xl/index.html\">Transformer XL</a> \u7684\u57fa\u7840\u4e0a\uff0c\u901a\u8fc7\u538b\u7f29\u6700\u65e9\u671f\u7684\u8bb0\u5fc6\u6765\u5ef6\u957f\u6ce8\u610f\u529b\u8de8\u5ea6\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/1911.00172\">Generalization through Memorization: Nearest Neighbor Language Models</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/1911.00172\">\u300a Generalization through Memorization: Nearest Neighbor Language Models \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2002.05202\">GLU Variants Improve Transformer</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587 <a href=\"https://arxiv.org/abs/2002.05202\">\u300a GLU Variants Improve Transformer \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2002.09402\">Accessing Higher-level Representations in Sequential Transformers with Feedback Memory</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/2002.09402\">\u300a Accessing Higher-level Representations in Sequential Transformers with Feedback Memory \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2010.11929\">An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/2010.11929\">\u300a An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2102.11174\">Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587 <a href=\"https://arxiv.org/abs/2102.11174\">\u300a Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.01601\">MLP-Mixer: An all-MLP Architecture for Vision</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587 <a href=\"https://arxiv.org/abs/2105.01601\">\u300a MLP-Mixer: An all-MLP Architecture for Vision \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/2105.03824\">\u300a FNet: Mixing Tokens with Fourier Transforms \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/2105.08050\">\u300a Pay Attention to MLPs \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2105.14103\">An Attention Free Transformer</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/2105.14103\">\u300a An Attention Free Transformer \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2109.08668\">Primer: Searching for Efficient Transformers for Language Modeling</a>.</p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/2109.08668\">\u300a Primer: Searching for Efficient Transformers for Language Modeling \u300b</a>\u7684\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>This is an implementation of the paper <a href=\"https://arxiv.org/abs/2110.13711\">Hierarchical Transformers Are More Efficient Language Models</a></p>\n": "<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/2110.13711\">\u300a Hierarchical Transformers Are More Efficient Language Models \u300b</a>\u7684\u5b9e\u73b0</p>\n",
|
||||
"<p>This module contains <a href=\"https://pytorch.org/\">PyTorch</a> implementations and explanations of original transformer from paper <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a>, and derivatives and enhancements of it.</p>\n": "</a><p>\u672c\u8282\u5185\u5bb9\u5305\u542b\u5bf9\u8bba\u6587<a href=\"https://arxiv.org/abs/1706.03762\">\u300a Attention is All You Need \u300b</a>\u4e2d\u539f\u59cb Transformer \u7684\u89e3\u91ca\u4e0e<a href=\"https://pytorch.org/\">PyTorch</a> \u5b9e\u73b0\uff0c\u4ee5\u53ca\u5bf9\u5176\u884d\u751f\u548c\u589e\u5f3a\u7248\u672c\u7684\u89e3\u91ca\u4e0e\u5b9e\u73b0\u3002</p>\n",
|
||||
"<ul><li><a href=\"mha.html\">Multi-head attention</a> </li>\n<li><a href=\"models.html\">Transformer Encoder and Decoder Models</a> </li>\n<li><a href=\"feed_forward.html\">Position-wise Feed Forward Network (FFN)</a> </li>\n<li><a href=\"positional_encoding.html\">Fixed positional encoding</a></li></ul>\n": "<ul><li><a href=\"mha.html\">\u591a\u5934\u6ce8\u610f\u529b</a></li>\n<li><a href=\"models.html\">Transformer \u7f16\u7801\u5668\u548c\u89e3\u7801\u5668\u6a21\u578b</a></li>\n<li><a href=\"feed_forward.html\">\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc (FFN)</a></li>\n<li><a href=\"positional_encoding.html\">\u56fa\u5b9a\u4f4d\u7f6e\u7f16\u7801</a></li></ul>\n",
|
||||
"This is a collection of PyTorch implementations/tutorials of transformers and related techniques.": "\u8fd9\u662f\u4e00\u4e2a\u5305\u542b Transformers \u53ca\u76f8\u5173\u6280\u672f\u7684 PyTorch \u5b9e\u73b0\u548c\u6559\u7a0b\u7684\u5408\u96c6\u3002",
|
||||
"Transformers": "Transformers"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Attention Free Transformer (AFT)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">AFT model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_autoregression.html\">general training loop and configurations for auto-regressive NLP task</a>.</p>\n": "<h1><a href=\"index.html\">\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d5\u30ea\u30fc\u5909\u5727\u5668\uff08AFT\uff09\u5b9f\u9a13</a></h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f\u3001AFT\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u306ePyTorch\u5b9f\u9a13\u3067\u3059\u3002</a></p>\n<p>\u3053\u308c\u306f\u3001<a href=\"../../experiments/nlp_autoregression.html\">\u4e00\u822c\u7684\u306a\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u81ea\u5df1\u56de\u5e30\u578bNLP\u30bf\u30b9\u30af\u306e\u8a2d\u5b9a\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u4ee5\u4e0b\u304b\u3089\u7d99\u627f\u3055\u308c\u307e\u3059 <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Simple autoregressive model</h2>\n<p>This consists of a token embedding layer, transformer encoder, and a final linear layer that gives token logits.</p>\n": "<h2>\u5358\u7d14\u306a\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n<p>\u3053\u308c\u306f\u3001\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u5c64\u3001\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u3001\u304a\u3088\u3073\u30c8\u30fc\u30af\u30f3\u30ed\u30b8\u30c3\u30c8\u3092\u63d0\u4f9b\u3059\u308b\u6700\u5f8c\u306e\u7dda\u5f62\u5c64\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create an auto-regressive model</p>\n": "<p>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u30de\u30b9\u30af\u304c\u521d\u671f\u5316\u3055\u308c\u3066\u3044\u306a\u3044\u5834\u5408\u3084\u30de\u30b9\u30af\u306e\u30b5\u30a4\u30ba\u304c\u7570\u306a\u308b\u5834\u5408\u306f\u3001\u5f8c\u7d9a\u306e\u30de\u30b9\u30af\u3092\u4f5c\u6210\u3057\u307e\u3059</p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>FFN hidden dimension size </p>\n": "<p>FFN \u96a0\u3057\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT \u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Optimizer </p>\n": "<p>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Replace self-attention with an <a href=\"index.html\">AFT Local Module</a> </p>\n": "<p><a href=\"index.html\">\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u3092AFT\u30ed\u30fc\u30ab\u30eb\u30e2\u30b8\u30e5\u30fc\u30eb\u306b\u7f6e\u304d\u63db\u3048\u308b</a></p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3057\u307e\u3059\uff08\u30c8\u30ec\u30fc\u30ca\u30fc\u306fRNN\u3067\u3082\u4f7f\u7528\u3055\u308c\u308b\u305f\u3081\u30012\u756a\u76ee\u306e\u5024\u306f\u72b6\u614b\u7528\u3067\u3059\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set the embedding size </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u6b21\u306b\u30de\u30b9\u30af\u3059\u308b\u3068\u3001\u30c8\u30fc\u30af\u30f3\u304c\u30de\u30b9\u30af\u3055\u308c\u3001\u5c06\u6765\u306e\u30c8\u30fc\u30af\u30f3\u304c\u898b\u3048\u306a\u304f\u306a\u308a\u307e\u3059</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u30de\u30b9\u30af\u306f\u6700\u521d\u306e\u547c\u3073\u51fa\u3057\u3067\u521d\u671f\u5316\u3055\u308c\u307e\u3059</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../models.html#Encoder\">\u5909\u5727\u5668\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span><a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u306f\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3059 (\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u4ed8\u304d)</a></li>\n</ul><li><span translate=no>_^_2_^_</span><a href=\"../models.html#Generator\">\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u6700\u5f8c\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u3067\u3059</a>\u3002</li>\n",
|
||||
"Attention Free Transformer (AFT) Experiment": "\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d5\u30ea\u30fc\u5909\u5727\u5668\uff08AFT\uff09\u5b9f\u9a13",
|
||||
"This experiment trains an Attention Free Transformer (AFT) based model on Tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u57fa\u3065\u3044\u3066\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30d5\u30ea\u30fc\u30fb\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\uff08AFT\uff09\u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Attention Free Transformer (AFT)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">AFT model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_autoregression.html\">general training loop and configurations for auto-regressive NLP task</a>.</p>\n<p><a href=\"https://app.labml.ai/run/6348e504c3a511eba9529daa283fb495\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db1\u0dd2\u0daf\u0dc4\u0dc3\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca (AFT)</a> \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">AFT \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf PyTorch \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p>\u0db8\u0dd9\u0dba <a href=\"../../experiments/nlp_autoregression.html\">\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda. </p>\n<p><a href=\"https://app.labml.ai/run/6348e504c3a511eba9529daa283fb495\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0db1\u0dca\u0db1\u0dda <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Simple autoregressive model</h2>\n<p>This consists of a token embedding layer, transformer encoder, and a final linear layer that gives token logits.</p>\n": "<h2>\u0dc3\u0dbb\u0dbd\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n<p>\u0db8\u0dd9\u0dba\u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dca, \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0dc4 \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0db1\u0dca\u0dc0\u0dd2\u0dad \u0dc0\u0dda. </p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create an auto-regressive model</p>\n": "<p> \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db1\u0db8\u0dca \u0dc4\u0ddd \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd9\u0dc4\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0db1\u0db8\u0dca \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>FFN hidden dimension size </p>\n": "<p>FFN\u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dda\u0dad\u0db1 \u0d9a\u0dca\u0dbb\u0db8 \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Optimizer </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Replace self-attention with an <a href=\"index.html\">AFT Local Module</a> </p>\n": "<p><a href=\"index.html\">AFT \u0daf\u0dda\u0dc1\u0dd3\u0dba \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba\u0d9a\u0dca</a> \u0dc3\u0db8\u0d9f \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0dcf\u0db7\u0db4\u0dca\u0dbb\u0dad\u0dd2 results \u0dbd (\u0daf\u0dd9\u0dc0\u0db1 \u0d85\u0d9c\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dda, \u0db8\u0db1\u0dca\u0daf \u0d85\u0db4\u0d9c\u0dda \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4 RNs \u0dc3\u0db8\u0d9f \u0daf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2) </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the embedding size </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab, \u0d85\u0db1\u0dcf\u0d9c\u0dad \u0da7\u0ddd\u0d9a\u0db1 \u0daf\u0dd0\u0d9a\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dc3\u0d82 \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0d87\u0db8\u0dad\u0dd4\u0db8\u0dd9\u0db1\u0dca \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad </p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"../models.html#Encoder\">\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a> </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0da7\u0ddd\u0d9a\u0db1\u0dca <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba (\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab \u0dc3\u0db8\u0d9f)</a> </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1 <a href=\"../models.html#Generator\">\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dba\u0dd2</a> . </li></ul>\n",
|
||||
"Attention Free Transformer (AFT) Experiment": "\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db1\u0dd2\u0daf\u0dc4\u0dc3\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca (AFT) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains an Attention Free Transformer (AFT) based model on Tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dbb\u0dc4\u0dd2\u0dad \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca (AFT) \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Attention Free Transformer (AFT)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">AFT model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_autoregression.html\">general training loop and configurations for auto-regressive NLP task</a>.</p>\n": "<h1><a href=\"index.html\">\u65e0\u6ce8\u610f\u53d8\u5f62\u91d1\u521a (AFT)</a> \u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3 A <a href=\"index.html\">FT \u6a21\u578b</a>\u3002</p>\n<p>\u8fd9\u662f\u57fa\u4e8e<a href=\"../../experiments/nlp_autoregression.html\">\u5e38\u89c4\u8bad\u7ec3\u5faa\u73af\u548c\u81ea\u56de\u5f52 NLP \u4efb\u52a1\u7684\u914d\u7f6e</a>\u3002</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Simple autoregressive model</h2>\n<p>This consists of a token embedding layer, transformer encoder, and a final linear layer that gives token logits.</p>\n": "<h2>\u7b80\u5355\u7684\u81ea\u56de\u5f52\u6a21\u578b</h2>\n<p>\u8fd9\u5305\u62ec\u4ee4\u724c\u5d4c\u5165\u5c42\u3001\u53d8\u538b\u5668\u7f16\u7801\u5668\u548c\u7ed9\u51fa\u4ee4\u724c\u65e5\u5fd7\u7684\u6700\u7ec8\u7ebf\u6027\u5c42\u3002</p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create an auto-regressive model</p>\n": "<p>\u521b\u5efa\u81ea\u52a8\u56de\u5f52\u6a21\u578b</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u5982\u679c\u63a9\u7801\u672a\u521d\u59cb\u5316\u6216\u63a9\u7801\u5927\u5c0f\u4e0d\u540c\uff0c\u5219\u521b\u5efa\u540e\u7eed\u63a9\u7801</p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>FFN hidden dimension size </p>\n": "<p>FFN \u9690\u85cf\u5c3a\u5bf8\u5927\u5c0f</p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT \u578b\u53f7</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u83b7\u53d6\u65e5\u5fd7</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f7f\u7528\u4f4d\u7f6e\u7f16\u7801\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Optimizer </p>\n": "<p>\u4f18\u5316\u5668</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Replace self-attention with an <a href=\"index.html\">AFT Local Module</a> </p>\n": "<p>\u7528 <a href=\"index.html\">AFT \u672c\u5730\u6a21\u5757</a>\u66ff\u6362\u81ea\u6211\u6ce8\u610f</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c\uff08\u7b2c\u4e8c\u4e2a\u503c\u7528\u4e8e\u72b6\u6001\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u8bad\u7ec3\u5668\u4e5f\u4e0e RNN \u4e00\u8d77\u4f7f\u7528\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set the embedding size </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u540e\u7eed\u7684\u63a9\u7801\uff0c\u5c06\u63a9\u76d6\u4ee4\u724c\u4ee5\u514d\u770b\u5230\u672a\u6765\u7684\u4ee3\u5e01</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u63a9\u7801\u5c06\u5728\u7b2c\u4e00\u6b21\u8c03\u7528\u65f6\u521d\u59cb\u5316</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668<a href=\"../models.html#Encoder\">\u7f16\u7801\u5668</a></li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c<a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u5d4c\u5165\u6a21\u5757\uff08\u5e26\u6709\u4f4d\u7f6e\u7f16\u7801\uff09</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u7ed9<a href=\"../models.html#Generator\">\u51fa logit \u7684\u6700\u540e\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42</a>\u3002</li></ul>\n",
|
||||
"Attention Free Transformer (AFT) Experiment": "\u514d\u6ce8\u610f\u53d8\u538b\u5668 (AFT) \u5b9e\u9a8c",
|
||||
"This experiment trains an Attention Free Transformer (AFT) based model on Tiny Shakespeare dataset.": "\u672c\u5b9e\u9a8c\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u57fa\u4e8e\u65e0\u6ce8\u610f\u529b\u53d8\u538b\u5668\uff08AFT\uff09\u7684\u6a21\u578b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/aft/index.html\">An Attention Free Transformer</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.14103\">An Attention Free Transformer</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers/mha.html\">self-attention layer</a> with a new efficient operation, that has memory complexity of O(Td), where T is the sequence length and <span translate=no>_^_0_^_</span> is the dimensionality of embeddings.</p>\n<p>The paper introduces AFT along with AFT-local and AFT-conv. Here we have implemented AFT-local which pays attention to closeby tokens in an autoregressive model. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/aft/index.html\">\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d5\u30ea\u30fc\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc</a></h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.14103\">\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30d5\u30ea\u30fc\u30fb\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<a href=\"https://nn.labml.ai/transformers/mha.html\">\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u5c64\u3092\u65b0\u3057\u3044\u52b9\u7387\u7684\u306a\u6f14\u7b97\u306b\u7f6e\u304d\u63db\u3048\u307e\u3059</a>\u3002\u30e1\u30e2\u30ea\u8907\u96d1\u5ea6\u306f O (Td) \u3067\u3001T \u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3067\u3001<span translate=no>_^_0_^_</span>\u57cb\u3081\u8fbc\u307f\u306e\u6b21\u5143\u3067\u3059\u3002</p>\n<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001AFT\u3068AFT\u30ed\u30fc\u30ab\u30eb\u304a\u3088\u3073AFT-Conv\u306b\u3064\u3044\u3066\u7d39\u4ecb\u3057\u3066\u3044\u307e\u3059\u3002\u3053\u3053\u3067\u306f\u3001\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u3067\u8fd1\u304f\u306e\u30c8\u30fc\u30af\u30f3\u306b\u6ce8\u76ee\u3059\u308bAFT-Local\u3092\u5b9f\u88c5\u3057\u307e\u3057\u305f</p>\u3002\n",
|
||||
"An Attention Free Transformer": "\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d5\u30ea\u30fc\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/aft/index.html\">An Attention Free Transformer</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.14103\">An Attention Free Transformer</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers/mha.html\">self-attention layer</a> with a new efficient operation, that has memory complexity of O(Td), where T is the sequence length and <span translate=no>_^_0_^_</span> is the dimensionality of embeddings.</p>\n<p>The paper introduces AFT along with AFT-local and AFT-conv. Here we have implemented AFT-local which pays attention to closeby tokens in an autoregressive model.</p>\n<p><a href=\"https://app.labml.ai/run/6348e504c3a511eba9529daa283fb495\"><span translate=no>_^_1_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/aft/index.html\">\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dbb\u0dc4\u0dd2\u0dad \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca</a></h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a <a href=\"https://arxiv.org/abs/2105.14103\">\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db1\u0dd2\u0daf\u0dc4\u0dc3\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a> . </p>\n<p>\u0db8\u0dd9\u0db8\u0dbd\u0dd2\u0db4\u0dd2\u0dba <a href=\"https://nn.labml.ai/transformers/mha.html\">\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dad\u0dbb\u0dba</a> \u0db1\u0dc0 \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8 \u0db8\u0dd9\u0dc4\u0dd9\u0dba\u0dd4\u0db8\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2, \u0d91\u0dba O (Td) \u0dc4\u0dd2 \u0db8\u0dad\u0d9a \u0dc3\u0d82\u0d9a\u0dd3\u0dbb\u0dca\u0dab\u0dad\u0dcf\u0dc0\u0dba\u0d9a\u0dca \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0da7\u0dd3 \u0dba\u0db1\u0dd4 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda \u0daf\u0dd2\u0d9c <span translate=no>_^_0_^_</span> \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc0\u0dda. </p>\n<p>\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2AFT \u0dc3\u0dc4 AFT \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0dba\u0dd2 AFT \u0dc3\u0dc4 AFT-conv. \u0db8\u0dd9\u0db1\u0dca\u0db1 \u0d85\u0db4\u0dd2 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dad\u0dd4\u0dc5 \u0dc3\u0db8\u0dd3\u0db4 \u0da7\u0ddd\u0d9a\u0db1 \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1 AFT- \u0daf\u0dda\u0dc1\u0dd3\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb \u0d87\u0dad\u0dca\u0dad\u0dd9\u0db8\u0dd4. </p>\n<p><a href=\"https://app.labml.ai/run/6348e504c3a511eba9529daa283fb495\"><span translate=no>_^_1_^_</span></a> </p>\n",
|
||||
"An Attention Free Transformer": "\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dbb\u0dc4\u0dd2\u0dad \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/aft/index.html\">An Attention Free Transformer</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.14103\">An Attention Free Transformer</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers/mha.html\">self-attention layer</a> with a new efficient operation, that has memory complexity of O(Td), where T is the sequence length and <span translate=no>_^_0_^_</span> is the dimensionality of embeddings.</p>\n<p>The paper introduces AFT along with AFT-local and AFT-conv. Here we have implemented AFT-local which pays attention to closeby tokens in an autoregressive model. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/aft/index.html\">\u4e00\u79cd\u65e0\u6ce8\u610f\u529b\u7684 Transformer </a></h1>\n<p>\u8fd9\u662f\u8bba\u6587 <a href=\"https://arxiv.org/abs/2105.14103\">\u300a\u4e00\u79cd\u65e0\u6ce8\u610f\u529b\u7684 Transformer \u300b</a>\u7684<a href=\"https://pytorch.org\">PyTorch </a>\u5b9e\u73b0\u3002</p>\n<p>\u8fd9\u7bc7\u8bba\u6587\u7528\u4e00\u79cd\u65b0\u7684\u9ad8\u6548\u64cd\u4f5c\u66ff\u4ee3\u4e86<a href=\"https://nn.labml.ai/transformers/mha.html\">\u81ea\u6ce8\u610f\u529b\u5c42</a>\uff0c\u8be5\u8fd0\u7b97\u7684\u5b58\u50a8\u590d\u6742\u5ea6\u4e3aO\uff08Td\uff09\uff0c\u5176\u4e2d T \u662f\u5e8f\u5217\u957f\u5ea6\uff0c<span translate=no>_^_0_^_</span>\u662f\u5d4c\u5165\u7684\u7ef4\u5ea6\u3002</p>\n<p>\u8be5\u8bba\u6587\u4ecb\u7ecd\u4e86 AFT \u4ee5\u53ca AFT-local \u548c AFT-conv \u3002\u8fd9\u91cc\u6211\u4eec\u5b9e\u73b0\u4e86 AFT-local \uff0c\u5b83\u4f1a\u5728\u81ea\u56de\u5f52\u6a21\u578b\u4e2d\u5173\u6ce8\u90bb\u8fd1\u7684 token \u3002</p>\n",
|
||||
"An Attention Free Transformer": "\u4e00\u79cd\u65e0\u6ce8\u610f\u529b\u7684 Transformer"
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"<h1>Attention with Linear Biases (ALiBi)</h1>\n<p>This is an implementation of Attention with Linear Biases (ALiBi) from the paper <a href=\"https://arxiv.org/abs/2108.12409\">Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation</a>.</p>\n<p>This replaces positional encodings with biases added to attention scores (attention logits, before the softmax). This is a relative scheme tested on autoregressive tasks, and the bias is higher for closeby tokens and lower for far-away tokens. The biases decrease linearly in the log scale (because it's before the softmax) and each head has a different slope.</p>\n<p>Here's the attention formula for <span translate=no>_^_0_^_</span>-th token,</p>\n<span translate=no>_^_1_^_</span><p>where <span translate=no>_^_2_^_</span> is the query of the <span translate=no>_^_3_^_</span>-th token, <span translate=no>_^_4_^_</span> are the keys up to <span translate=no>_^_5_^_</span>, and <span translate=no>_^_6_^_</span> the number of features per head. Note that the above equality halts because <span translate=no>_^_7_^_</span> is invariant to translations (you can add any constant to all elements without changing the result).</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a ALiBi model.</p>\n": "<h1>\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f (AliBi)</h1>\n<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/2108.12409\">\u30c8\u30ec\u30a4\u30f3\u30b7\u30e7\u30fc\u30c8\u3001\u30c6\u30b9\u30c8\u30ed\u30f3\u30b0\u300d\u3068\u3044\u3046\u8ad6\u6587\u306e\u300c\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f\uff08AliBi\uff09\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f\u306b\u3088\u308a\u3001\u5165\u529b\u306e\u9577\u3055\u306e\u63a8\u5b9a\u304c\u53ef\u80fd\u306b\u306a\u308a\u307e\u3059</a>\u3002</p>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u304c\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2\uff08\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9\u306e\u524d\u306e\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ed\u30b8\u30c3\u30c8\uff09\u306b\u30d0\u30a4\u30a2\u30b9\u304c\u52a0\u308f\u3063\u305f\u3082\u306e\u306b\u7f6e\u304d\u63db\u308f\u308a\u307e\u3059\u3002\u3053\u308c\u306f\u81ea\u5df1\u56de\u5e30\u30bf\u30b9\u30af\u3067\u30c6\u30b9\u30c8\u3055\u308c\u305f\u76f8\u5bfe\u7684\u306a\u30b9\u30ad\u30fc\u30e0\u3067\u3001\u8fd1\u304f\u306b\u3042\u308b\u30c8\u30fc\u30af\u30f3\u306e\u65b9\u304c\u30d0\u30a4\u30a2\u30b9\u304c\u5927\u304d\u304f\u3001\u9060\u3044\u30c8\u30fc\u30af\u30f3\u306e\u65b9\u304c\u30d0\u30a4\u30a2\u30b9\u304c\u4f4e\u304f\u306a\u308a\u307e\u3059\u3002\u5bfe\u6570\u30b9\u30b1\u30fc\u30eb\u3067\u306f\uff08\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9\u306e\u524d\u306a\u306e\u3067\uff09\u30d0\u30a4\u30a2\u30b9\u306f\u76f4\u7dda\u7684\u306b\u6e1b\u5c11\u3057\u3001\u5404\u30d8\u30c3\u30c9\u306e\u50be\u304d\u306f\u7570\u306a\u308a\u307e\u3059</p>\u3002\n<p><span translate=no>_^_0_^_</span>-th \u30c8\u30fc\u30af\u30f3\u306e\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d5\u30a9\u30fc\u30df\u30e5\u30e9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059\u3002</p>\n<span translate=no>_^_1_^_</span><p>\u3053\u3053\u3067\u3001<span translate=no>_^_2_^_</span>\u306f <span translate=no>_^_3_^_</span>-th \u30c8\u30fc\u30af\u30f3\u306e\u30af\u30a8\u30ea\u3001<span translate=no>_^_4_^_</span>\u307e\u3067\u306e\u30ad\u30fc<span translate=no>_^_5_^_</span>\u3001<span translate=no>_^_6_^_</span>\u304a\u3088\u3073\u30d8\u30c3\u30c9\u3042\u305f\u308a\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u6570\u3067\u3059\u3002<span translate=no>_^_7_^_</span>\u4e0a\u8a18\u306e\u7b49\u5f0f\u306f\u5909\u63db\u306b\u4e0d\u5909\u3067\u3042\u308b\u305f\u3081\u4e2d\u6b62\u3055\u308c\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044 (\u7d50\u679c\u3092\u5909\u66f4\u305b\u305a\u306b\u3059\u3079\u3066\u306e\u8981\u7d20\u306b\u4efb\u610f\u306e\u5b9a\u6570\u3092\u8ffd\u52a0\u3067\u304d\u307e\u3059</p>)\u3002\n<p>AliBi <a href=\"experiment.html\">\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059</a>\u3002</p>\n",
|
||||
"<h2>Attention with Linear Biases (ALiBi)</h2>\n<p>We override <a href=\"../mha.html\">Multi-Head Attention</a>.</p>\n": "<h2>\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f (AliBi)</h2>\n<p><a href=\"../mha.html\">\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u3092\u7121\u52b9\u306b\u3057\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h2>Calculate the attention biases matrix</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the number of heads in the attention layer </li>\n<li><span translate=no>_^_1_^_</span> is the attention mask of shape <span translate=no>_^_2_^_</span></li></ul>\n<p>This returns a matrix of shape <span translate=no>_^_3_^_</span> with ALiBi attention biases.</p>\n": "<h2>\u6ce8\u610f\u30d0\u30a4\u30a2\u30b9\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u306e\u8a08\u7b97</h2>\n<ul><li><span translate=no>_^_0_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u306e\u30d8\u30c3\u30c9\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30b7\u30a7\u30a4\u30d7\u306e\u6ce8\u610f\u30de\u30b9\u30af\u3067\u3059 <span translate=no>_^_2_^_</span></li></ul>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001AliBi <span translate=no>_^_3_^_</span> \u306e\u6ce8\u610f\u30d0\u30a4\u30a2\u30b9\u304c\u5165\u3063\u305f\u5f62\u72b6\u306e\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u304c\u8fd4\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<h2>Get head-specific slope <span translate=no>_^_0_^_</span> for each head</h2>\n<ul><li><span translate=no>_^_1_^_</span> is the number of heads in the attention layer <span translate=no>_^_2_^_</span></li></ul>\n<p>The slope for first head is</p>\n<p><span translate=no>_^_3_^_</span></p>\n<p>The slopes for the rest of the heads are in a geometric series with a ratio same as above.</p>\n<p>For instance when the number of heads is <span translate=no>_^_4_^_</span> the slopes are <span translate=no>_^_5_^_</span></p>\n": "<h2><span translate=no>_^_0_^_</span>\u5404\u982d\u90e8\u306e\u982d\u90e8\u56fa\u6709\u306e\u52fe\u914d\u3092\u53d6\u5f97</h2>\n<ul><li><span translate=no>_^_1_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u306e\u30d8\u30c3\u30c9\u6570\u3067\u3059 <span translate=no>_^_2_^_</span></li></ul>\n<p>1 \u756a\u76ee\u306e\u30d8\u30c3\u30c9\u306e\u52fe\u914d\u306f</p>\n<p><span translate=no>_^_3_^_</span></p>\n<p>\u6b8b\u308a\u306e\u30d8\u30c3\u30c9\u306e\u52fe\u914d\u306f\u5e7e\u4f55\u5b66\u7684\u306b\u9023\u7d9a\u3057\u3066\u304a\u308a\u3001\u305d\u306e\u6bd4\u7387\u306f\u4e0a\u8a18\u3068\u540c\u3058\u3067\u3059\u3002</p>\n<p>\u305f\u3068\u3048\u3070\u3001\u30d8\u30c3\u30c9\u306e\u6570\u304c\u306e\u5834\u5408\u3001<span translate=no>_^_4_^_</span>\u30b9\u30ed\u30fc\u30d7\u306f <span translate=no>_^_5_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> are the tensors that store collection of <em>query</em>, <em>key</em> and <em>value</em> vectors. They have shape <span translate=no>_^_3_^_</span>.</p>\n<p><span translate=no>_^_4_^_</span> has shape <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> indicates whether for batch <span translate=no>_^_7_^_</span>, query at position <span translate=no>_^_8_^_</span> has access to key-value at position <span translate=no>_^_9_^_</span>.</p>\n": "<p><span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u304a\u3088\u3073\u306f\u3001<em>\u30af\u30a8\u30ea</em>\u3001<em>\u30ad\u30fc</em>\u3001<em>\u304a\u3088\u3073\u5024\u306e\u30d9\u30af\u30c8\u30eb\u306e\u30b3\u30ec\u30af\u30b7\u30e7\u30f3\u3092\u683c\u7d0d\u3059\u308b\u30c6\u30f3\u30bd\u30eb\u3067\u3059</em>\u3002\u5f62\u304c\u3042\u308a\u307e\u3059<span translate=no>_^_3_^_</span>\u3002</p>\n<p><span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span>\u5f62\u72b6\u304c\u3042\u308a\u3001\u30d0\u30c3\u30c1\u306e\u5834\u5408<span translate=no>_^_7_^_</span>\u3001<span translate=no>_^_6_^_</span><span translate=no>_^_8_^_</span>\u305d\u306e\u4f4d\u7f6e\u306e\u30af\u30a8\u30ea\u304c\u305d\u306e\u4f4d\u7f6e\u306e\u30ad\u30fc\u5024\u306b\u30a2\u30af\u30bb\u30b9\u3067\u304d\u308b\u304b\u3069\u3046\u304b\u3092\u793a\u3057\u307e\u3059\u3002<span translate=no>_^_9_^_</span></p>\n",
|
||||
"<p> Simple test function to see the slopes.</p>\n": "<p>\u30b9\u30ed\u30fc\u30d7\u3092\u78ba\u8a8d\u3067\u304d\u308b\u7c21\u5358\u306a\u30c6\u30b9\u30c8\u6a5f\u80fd\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> Note that we take steps by <span translate=no>_^_1_^_</span> to avoid slopes added previously. </p>\n": "<p><span translate=no>_^_0_^_</span>\u306a\u304a\u3001<span translate=no>_^_1_^_</span>\u4ee5\u524d\u306b\u30b9\u30ed\u30fc\u30d7\u304c\u8ffd\u52a0\u3055\u308c\u306a\u3044\u3088\u3046\u306b\u5bfe\u7b56\u3092\u8b1b\u3058\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> attention along the key sequence dimension <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u30ad\u30fc\u30b7\u30fc\u30b1\u30f3\u30b9\u6b21\u5143\u306b\u6cbf\u3063\u3066\u6ce8\u76ee <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> has shape <a href=\"seq_len, seq_len, 1, 1\">seq_len, seq_len, 1, 1</a> </p>\n": "<p><span translate=no>_^_0_^_</span><a href=\"seq_len, seq_len, 1, 1\">\u56f3\u5f62\u306f\u9023\u756a\u3001\u9023\u756a\u30011\u3001</a> 1\u3067\u3059</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> have shape <span translate=no>_^_3_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u305d\u3057\u3066\u5f62\u304c\u3042\u308b <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p>ALiBi only works with causal masks. </p>\n": "<p>AliBi \u306f\u56e0\u679c\u30de\u30b9\u30af\u3067\u306e\u307f\u6a5f\u80fd\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Add AliBi biases to attention scores. ALiBi biases has shape <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> has shape <span translate=no>_^_2_^_</span> </p>\n": "<p>AliBi \u30d0\u30a4\u30a2\u30b9\u3092\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2\u306b\u8ffd\u52a0\u3057\u307e\u3059\u3002AliBi <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u30d0\u30a4\u30a2\u30b9\u306b\u306f\u5f62\u3068\u5f62\u304c\u3042\u308b <span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>Add head dimension to mask and check its shape. </p>\n": "<p>\u30de\u30b9\u30af\u306b\u982d\u90e8\u306e\u5bf8\u6cd5\u3092\u8ffd\u52a0\u3057\u3001\u5f62\u72b6\u3092\u78ba\u8a8d\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Apply dropout </p>\n": "<p>\u30c9\u30ed\u30c3\u30d7\u30a2\u30a6\u30c8\u3092\u9069\u7528</p>\n",
|
||||
"<p>Apply mask </p>\n": "<p>\u30de\u30b9\u30af\u3092\u9069\u7528</p>\n",
|
||||
"<p>Calculate distances <span translate=no>_^_0_^_</span> Here we calculate the distances using the mask.</p>\n<p>Since it's causal mask we can just use <span translate=no>_^_1_^_</span> too. <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u8ddd\u96e2\u306e\u8a08\u7b97\u3053\u3053\u3067\u306f\u30de\u30b9\u30af\u3092\u4f7f\u3063\u3066\u8ddd\u96e2\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002</p>\n<p><span translate=no>_^_1_^_</span>\u30ab\u30b8\u30e5\u30a2\u30eb\u30de\u30b9\u30af\u306a\u306e\u3067\u305d\u306e\u307e\u307e\u4f7f\u3048\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>Compute attention scores <span translate=no>_^_0_^_</span>. This gives a tensor of shape <span translate=no>_^_1_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002<span translate=no>_^_1_^_</span>\u3053\u308c\u306b\u3088\u308a\u5f62\u72b6\u306e\u30c6\u30f3\u30bd\u30eb\u304c\u5f97\u3089\u308c\u307e\u3059</p>\u3002\n",
|
||||
"<p>Concatenate multiple heads </p>\n": "<p>\u8907\u6570\u306e\u30d8\u30c3\u30c9\u3092\u9023\u7d50</p>\n",
|
||||
"<p>Concatenate the slopes with the remaining slopes. </p>\n": "<p>\u30b9\u30ed\u30fc\u30d7\u3092\u6b8b\u308a\u306e\u30b9\u30ed\u30fc\u30d7\u3068\u9023\u7d50\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Create AliBi biases if it's not cached </p>\n": "<p>\u30ad\u30e3\u30c3\u30b7\u30e5\u3055\u308c\u3066\u3044\u306a\u3044\u5834\u5408\u306fAliBi\u30d0\u30a4\u30a2\u30b9\u3092\u4f5c\u6210\u3059\u308b</p>\n",
|
||||
"<p>Get slopes <span translate=no>_^_0_^_</span> for each head </p>\n": "<p><span translate=no>_^_0_^_</span>\u5404\u30d8\u30c3\u30c9\u306e\u30b9\u30ed\u30fc\u30d7\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the closest power of 2 to <span translate=no>_^_0_^_</span>. If <span translate=no>_^_1_^_</span> is not a power of 2, then we first calculate slopes to the closest (smaller) power of 2, and then add the remaining slopes. </p>\n": "<p>2 <span translate=no>_^_0_^_</span> \u306e\u7d2f\u4e57\u306b\u6700\u3082\u8fd1\u3044\u3082\u306e\u3092\u6c42\u3081\u307e\u3059\u3002\u304c 2 <span translate=no>_^_1_^_</span> \u306e\u7d2f\u4e57\u3067\u306a\u3044\u5834\u5408\u306f\u3001\u307e\u305a 2 \u306b\u6700\u3082\u8fd1\u3044 (\u5c0f\u3055\u306a) \u7d2f\u4e57\u307e\u3067\u306e\u52fe\u914d\u3092\u8a08\u7b97\u3057\u3001\u6b21\u306b\u6b8b\u308a\u306e\u52fe\u914d\u3092\u52a0\u7b97\u3057\u307e\u3059</p>\u3002\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> is not a power of 2, then we add the remaining slopes. We calculate the remaining slopes for <span translate=no>_^_1_^_</span> (avoiding slopes added previously). And pick the slopes upto <span translate=no>_^_2_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span>\u304c 2 \u306e\u7d2f\u4e57\u3067\u306a\u3044\u5834\u5408\u306f\u3001\u6b8b\u308a\u306e\u52fe\u914d\u3092\u52a0\u7b97\u3057\u307e\u3059\u3002\u6b8b\u308a\u306e\u52fe\u914d\u3092\u8a08\u7b97\u3057\u307e\u3059 <span translate=no>_^_1_^_</span> (\u4ee5\u524d\u306b\u8ffd\u52a0\u3055\u308c\u305f\u52fe\u914d\u306f\u9664\u304d\u307e\u3059)\u3002\u305d\u3057\u3066\u3001<span translate=no>_^_2_^_</span>\u4e0a\u306e\u659c\u9762\u3092\u9078\u3093\u3067\u304f\u3060\u3055\u3044</p>.\n",
|
||||
"<p>Multiply by values <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5024\u306b\u3088\u308b\u4e57\u7b97 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Multiply them pair-wise to get the AliBi bias matrix </p>\n": "<p>\u305d\u308c\u3089\u3092\u30da\u30a2\u3054\u3068\u306b\u4e57\u7b97\u3057\u3066\u3001AliBi \u30d0\u30a4\u30a2\u30b9\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u3092\u6c42\u3081\u307e\u3059\u3002</p>\n",
|
||||
"<p>Output layer </p>\n": "<p>\u51fa\u529b\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Prepare <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> for attention computation. These will then have shape <span translate=no>_^_3_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u6ce8\u610f\u529b\u8a08\u7b97\u306e\u6e96\u5099\u3092\u3057\u3066<span translate=no>_^_3_^_</span>\u3053\u308c\u3067\u5f62\u304c\u3067\u304d\u3042\u304c\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<p>Scale scores <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30b1\u30fc\u30eb\u30b9\u30b3\u30a2 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>To cache AliBi the biases </p>\n": "<p>AliBi \u306b\u30d0\u30a4\u30a2\u30b9\u3092\u30ad\u30e3\u30c3\u30b7\u30e5\u3059\u308b\u306b\u306f</p>\n",
|
||||
"Attention with Linear Biases (ALiBi)": "\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f (AliBi)",
|
||||
"Documented implementation with explanations of Attention with Linear Biases (ALiBi)": "\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f\uff08AliBi\uff09\u306e\u8aac\u660e\u3092\u542b\u3080\u6587\u66f8\u5316\u3055\u308c\u305f\u5b9f\u88c5"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"<h1>Attention with Linear Biases (ALiBi)</h1>\n<p>This is an implementation of Attention with Linear Biases (ALiBi) from the paper <a href=\"https://arxiv.org/abs/2108.12409\">Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation</a>.</p>\n<p>This replaces positional encodings with biases added to attention scores (attention logits, before the softmax). This is a relative scheme tested on autoregressive tasks, and the bias is higher for closeby tokens and lower for far-away tokens. The biases decrease linearly in the log scale (because it's before the softmax) and each head has a different slope.</p>\n<p>Here's the attention formula for <span translate=no>_^_0_^_</span>-th token,</p>\n<span translate=no>_^_1_^_</span><p>where <span translate=no>_^_2_^_</span> is the query of the <span translate=no>_^_3_^_</span>-th token, <span translate=no>_^_4_^_</span> are the keys up to <span translate=no>_^_5_^_</span>, and <span translate=no>_^_6_^_</span> the number of features per head. Note that the above equality halts because <span translate=no>_^_7_^_</span> is invariant to translations (you can add any constant to all elements without changing the result).</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a ALiBi model.</p>\n": "<h1>\u6ce8\u610f\u7ebf\u6027\u504f\u5dee (AliBI)</h1>\n<p>\u8fd9\u662f\u300aT <a href=\"https://arxiv.org/abs/2108.12409\">rain Short\uff0cTest Long\uff1a\u4f7f\u7528\u7ebf\u6027\u504f\u5dee\u7684\u6ce8\u610f\u529b\u5b9e\u73b0\u8f93\u5165\u957f\u5ea6\u5916\u63a8\u300b\u4e00\u6587\u4e2d\u7684 \u201c\u4f7f\u7528\u7ebf\u6027\u504f\u5dee\u6ce8\u610f\u529b</a> (AliBI)\u201d \u7684\u5b9e\u73b0\u3002</p>\n<p>\u8fd9\u5c06\u7528\u5728\u6ce8\u610f\u529b\u5206\u6570\u4e2d\u6dfb\u52a0\u504f\u5dee\uff08\u6ce8\u610f\u529b\u5bf9\u6570\uff0c\u5728 softmax \u4e4b\u524d\uff09\u53d6\u4ee3\u4f4d\u7f6e\u7f16\u7801\u3002\u8fd9\u662f\u4e00\u79cd\u5728\u81ea\u56de\u5f52\u4efb\u52a1\u4e0a\u6d4b\u8bd5\u7684\u76f8\u5bf9\u65b9\u6848\uff0ccloseby\u4ee3\u5e01\u7684\u504f\u5dee\u66f4\u9ad8\uff0c\u800c\u9065\u8fdc\u7684\u4ee3\u5e01\u7684\u504f\u5dee\u66f4\u4f4e\u3002\u504f\u5dee\u5728\u5bf9\u6570\u6807\u5ea6\u4e2d\u5448\u7ebf\u6027\u51cf\u5c0f\uff08\u56e0\u4e3a\u5b83\u5728softmax\u4e4b\u524d\uff09\uff0c\u5e76\u4e14\u6bcf\u4e2a\u5934\u90e8\u90fd\u6709\u4e0d\u540c\u7684\u659c\u7387\u3002</p>\n<p>\u8fd9\u662f<span translate=no>_^_0_^_</span>\u7b2c-th \u4ee3\u5e01\u7684\u6ce8\u610f\u529b\u516c\u5f0f\uff0c</p>\n<span translate=no>_^_1_^_</span><p>\u5176\u4e2d\uff0c<span translate=no>_^_2_^_</span>\u662f<span translate=no>_^_3_^_</span>\u7b2c-th \u4e2a\u4ee4\u724c\u7684\u67e5\u8be2\uff0c\u6700\u5927<span translate=no>_^_4_^_</span>\u662f\u5bc6\u94a5\u6570\u4ee5\u53ca<span translate=no>_^_6_^_</span>\u6bcf\u4e2a\u6807\u5934\u7684\u8981<span translate=no>_^_5_^_</span>\u7d20\u6570\u3002\u8bf7\u6ce8\u610f\uff0c\u4e0a\u8ff0\u7b49\u5f0f\u4e4b\u6240\u4ee5\u505c\u6b62\uff0c\u662f\u56e0\u4e3a\u7ffb\u8bd1\u662f\u4e0d\u53d8<span translate=no>_^_7_^_</span>\u7684\uff08\u60a8\u53ef\u4ee5\u5728\u4e0d\u66f4\u6539\u7ed3\u679c\u7684\u60c5\u51b5\u4e0b\u5411\u6240\u6709\u5143\u7d20\u6dfb\u52a0\u4efb\u4f55\u5e38\u91cf\uff09\u3002</p>\n<p><a href=\"experiment.html\">\u4ee5\u4e0b\u662f AliBi \u6a21\u578b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
|
||||
"<h2>Attention with Linear Biases (ALiBi)</h2>\n<p>We override <a href=\"../mha.html\">Multi-Head Attention</a>.</p>\n": "<h2>\u6ce8\u610f\u7ebf\u6027\u504f\u5dee (AliBI)</h2>\n<p>\u6211\u4eec\u8986\u76d6<a href=\"../mha.html\">\u591a\u5934\u6ce8\u610f\u529b</a>\u3002</p>\n",
|
||||
"<h2>Calculate the attention biases matrix</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the number of heads in the attention layer </li>\n<li><span translate=no>_^_1_^_</span> is the attention mask of shape <span translate=no>_^_2_^_</span></li></ul>\n<p>This returns a matrix of shape <span translate=no>_^_3_^_</span> with ALiBi attention biases.</p>\n": "<h2>\u8ba1\u7b97\u6ce8\u610f\u529b\u504f\u5dee\u77e9\u9635</h2>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u6ce8\u610f\u5c42\u4e2d\u7684\u5934\u90e8\u6570\u91cf</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5f62\u72b6\u7684\u6ce8\u610f\u529b\u9762\u5177<span translate=no>_^_2_^_</span></li></ul>\n<p>\u8fd9\u5c06\u8fd4\u56de\u4e00\u4e2a<span translate=no>_^_3_^_</span>\u5177\u6709 AliBi \u6ce8\u610f\u529b\u504f\u5dee\u7684\u5f62\u72b6\u77e9\u9635\u3002</p>\n",
|
||||
"<h2>Get head-specific slope <span translate=no>_^_0_^_</span> for each head</h2>\n<ul><li><span translate=no>_^_1_^_</span> is the number of heads in the attention layer <span translate=no>_^_2_^_</span></li></ul>\n<p>The slope for first head is</p>\n<p><span translate=no>_^_3_^_</span></p>\n<p>The slopes for the rest of the heads are in a geometric series with a ratio same as above.</p>\n<p>For instance when the number of heads is <span translate=no>_^_4_^_</span> the slopes are <span translate=no>_^_5_^_</span></p>\n": "<h2><span translate=no>_^_0_^_</span>\u4e3a\u6bcf\u4e2a\u5934\u90e8\u83b7\u53d6\u7279\u5b9a\u4e8e\u5934\u90e8\u7684\u659c\u7387</h2>\n<ul><li><span translate=no>_^_1_^_</span>\u662f\u6ce8\u610f\u5c42\u4e2d\u7684\u5934\u90e8\u6570\u91cf<span translate=no>_^_2_^_</span></li></ul>\n<p>\u7b2c\u4e00\u4e2a\u5934\u7684\u659c\u7387\u662f</p>\n<p><span translate=no>_^_3_^_</span></p>\n<p>\u5176\u4f59\u5934\u90e8\u7684\u659c\u7387\u4e3a\u51e0\u4f55\u5e8f\u5217\uff0c\u5176\u6bd4\u4f8b\u4e0e\u4e0a\u9762\u76f8\u540c\u3002</p>\n<p>\u4f8b\u5982\uff0c\u5f53\u5934\u6570\u4e3a\u65f6<span translate=no>_^_4_^_</span>\uff0c\u659c\u7387\u4e3a<span translate=no>_^_5_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> are the tensors that store collection of <em>query</em>, <em>key</em> and <em>value</em> vectors. They have shape <span translate=no>_^_3_^_</span>.</p>\n<p><span translate=no>_^_4_^_</span> has shape <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> indicates whether for batch <span translate=no>_^_7_^_</span>, query at position <span translate=no>_^_8_^_</span> has access to key-value at position <span translate=no>_^_9_^_</span>.</p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u548c<span translate=no>_^_2_^_</span>\u662f\u5b58\u50a8<em>\u67e5\u8be2</em>\u3001<em>\u952e</em>\u548c<em>\u503c</em>\u5411\u91cf\u96c6\u5408\u7684\u5f20\u91cf\u3002\u5b83\u4eec\u6709\u5f62\u72b6<span translate=no>_^_3_^_</span>\u3002</p>\n<p><span translate=no>_^_4_^_</span>\u6709\u5f62\u72b6<span translate=no>_^_5_^_</span>\u5e76<span translate=no>_^_6_^_</span>\u6307\u793a\u662f\u5426\u4e3a\u6279\u91cf\u67e5\u8be2<span translate=no>_^_7_^_</span>\uff0c\u4f4d\u7f6e\u5904\u7684\u67e5\u8be2<span translate=no>_^_8_^_</span>\u6709\u6743\u8bbf\u95ee\u4f4d\u7f6e\u5904\u7684\u952e\u503c<span translate=no>_^_9_^_</span>\u3002</p>\n",
|
||||
"<p> Simple test function to see the slopes.</p>\n": "<p>\u67e5\u770b\u659c\u7387\u7684\u7b80\u5355\u6d4b\u8bd5\u529f\u80fd\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> Note that we take steps by <span translate=no>_^_1_^_</span> to avoid slopes added previously. </p>\n": "<p><span translate=no>_^_0_^_</span>\u8bf7\u6ce8\u610f\uff0c\u6211\u4eec\u4f1a\u91c7\u53d6\u63aa\u65bd<span translate=no>_^_1_^_</span>\u907f\u514d\u4e4b\u524d\u6dfb\u52a0\u7684\u659c\u5761\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> attention along the key sequence dimension <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u5173\u6ce8\u6309\u952e\u5e8f\u5217\u7ef4\u5ea6<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> have shape <span translate=no>_^_3_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\uff0c<span translate=no>_^_1_^_</span>\u5e76\u4e14<span translate=no>_^_2_^_</span>\u6709\u5f62\u72b6<span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p>ALiBi only works with causal masks. </p>\n": "<p>AliBi \u4ec5\u9002\u7528\u4e8e\u56e0\u679c\u53e3\u7f69\u3002</p>\n",
|
||||
"<p>Add AliBi biases to attention scores. ALiBi biases has shape <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> has shape <span translate=no>_^_2_^_</span> </p>\n": "<p>\u5c06 AliBi \u504f\u89c1\u6dfb\u52a0\u5230\u6ce8\u610f\u529b\u5206\u6570\u4e2d\u3002AliBi \u504f\u89c1\u6709\u5f62<span translate=no>_^_0_^_</span>\u72b6<span translate=no>_^_1_^_</span>\u4e5f\u6709\u5f62\u72b6<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>Add head dimension to mask and check its shape. </p>\n": "<p>\u5c06\u5934\u90e8\u5c3a\u5bf8\u6dfb\u52a0\u5230\u8499\u7248\u5e76\u68c0\u67e5\u5176\u5f62\u72b6\u3002</p>\n",
|
||||
"<p>Apply dropout </p>\n": "<p>\u7533\u8bf7\u9000\u5b66</p>\n",
|
||||
"<p>Apply mask </p>\n": "<p>\u6d82\u62b9\u9762\u819c</p>\n",
|
||||
"<p>Calculate distances <span translate=no>_^_0_^_</span> Here we calculate the distances using the mask.</p>\n<p>Since it's causal mask we can just use <span translate=no>_^_1_^_</span> too. <span translate=no>_^_2_^_</span> </p>\n": "<p>\u8ba1\u7b97\u8ddd\u79bb<span translate=no>_^_0_^_</span>\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528\u63a9\u7801\u8ba1\u7b97\u8ddd\u79bb\u3002</p>\n<p>\u65e2\u7136\u5b83\u662f\u56e0\u679c\u63a9\u7801\uff0c\u6211\u4eec<span translate=no>_^_1_^_</span>\u4e5f\u53ef\u4ee5\u4f7f\u7528\u3002<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>Compute attention scores <span translate=no>_^_0_^_</span>. This gives a tensor of shape <span translate=no>_^_1_^_</span>. </p>\n": "<p>\u8ba1\u7b97\u6ce8\u610f\u529b\u5206\u6570<span translate=no>_^_0_^_</span>\u3002\u8fd9\u7ed9\u51fa\u4e86\u5f62\u72b6\u7684\u5f20\u91cf<span translate=no>_^_1_^_</span>\u3002</p>\n",
|
||||
"<p>Concatenate multiple heads </p>\n": "<p>\u8fde\u63a5\u591a\u4e2a\u5934</p>\n",
|
||||
"<p>Concatenate the slopes with the remaining slopes. </p>\n": "<p>\u5c06\u659c\u5761\u4e0e\u5176\u4f59\u7684\u659c\u5761\u8fde\u63a5\u8d77\u6765\u3002</p>\n",
|
||||
"<p>Create AliBi biases if it's not cached </p>\n": "<p>\u5982\u679c AliBI \u672a\u88ab\u7f13\u5b58\uff0c\u5219\u521b\u5efa\u504f\u5dee</p>\n",
|
||||
"<p>Get slopes <span translate=no>_^_0_^_</span> for each head </p>\n": "<p>\u83b7\u53d6\u6bcf\u4e2a<span translate=no>_^_0_^_</span>\u5934\u90e8\u7684\u659c\u7387</p>\n",
|
||||
"<p>Get the closest power of 2 to <span translate=no>_^_0_^_</span>. If <span translate=no>_^_1_^_</span> is not a power of 2, then we first calculate slopes to the closest (smaller) power of 2, and then add the remaining slopes. </p>\n": "<p>\u83b7\u5f97\u6700\u63a5\u8fd1 2 \u7684\u5e42<span translate=no>_^_0_^_</span>\u3002\u5982\u679c\u4e0d<span translate=no>_^_1_^_</span>\u662f 2 \u7684\u5e42\uff0c\u90a3\u4e48\u6211\u4eec\u9996\u5148\u8ba1\u7b97\u659c\u7387\u5230\u6700\u63a5\u8fd1\uff08\u8f83\u5c0f\uff09\u7684 2 \u5e42\uff0c\u7136\u540e\u518d\u52a0\u4e0a\u5269\u4f59\u7684\u659c\u7387\u3002</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> is not a power of 2, then we add the remaining slopes. We calculate the remaining slopes for <span translate=no>_^_1_^_</span> (avoiding slopes added previously). And pick the slopes upto <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u5982\u679c\u4e0d<span translate=no>_^_0_^_</span>\u662f 2 \u7684\u5e42\uff0c\u90a3\u4e48\u6211\u4eec\u5c06\u5269\u4f59\u7684\u659c\u7387\u76f8\u52a0\u3002\u6211\u4eec\u8ba1\u7b97\u5269\u4f59\u7684\u659c\u7387<span translate=no>_^_1_^_</span>\uff08\u907f\u514d\u4e4b\u524d\u6dfb\u52a0\u7684\u659c\u7387\uff09\u3002\u7136\u540e\u9009\u62e9\u659c\u5761<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"<p>Multiply by values <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4e58\u4ee5\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Multiply them pair-wise to get the AliBi bias matrix </p>\n": "<p>\u5c06\u5b83\u4eec\u6210\u5bf9\u4e58\u4ee5\u5f97\u5230 AliBi \u504f\u5dee\u77e9\u9635</p>\n",
|
||||
"<p>Output layer </p>\n": "<p>\u8f93\u51fa\u5c42</p>\n",
|
||||
"<p>Prepare <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> for attention computation. These will then have shape <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u51c6\u5907<span translate=no>_^_0_^_</span>\uff0c<span translate=no>_^_1_^_</span>\u5e76<span translate=no>_^_2_^_</span>\u8fdb\u884c\u6ce8\u610f\u529b\u8ba1\u7b97\u3002\u7136\u540e\u8fd9\u4e9b\u5c31\u4f1a\u6709\u5f62\u72b6<span translate=no>_^_3_^_</span>\u3002</p>\n",
|
||||
"<p>Scale scores <span translate=no>_^_0_^_</span> </p>\n": "<p>\u97f3\u9636\u5206\u6570<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>To cache AliBi the biases </p>\n": "<p>\u7f13\u5b58 AliBi \u7684\u504f\u89c1</p>\n",
|
||||
"Attention with Linear Biases (ALiBi)": "\u6ce8\u610f\u7ebf\u6027\u504f\u5dee (AliBI)",
|
||||
"Documented implementation with explanations of Attention with Linear Biases (ALiBi)": "\u8bb0\u5f55\u5b9e\u73b0\uff0c\u5e76\u89e3\u91ca\u7ebf\u6027\u504f\u5dee\u6ce8\u610f\u529b (AliBi)"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Attention with Linear Biases (ALiBi)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">ALiBi model</a>.</p>\n<p>This is based on <a href=\"../gpt/index.html\">our GPT model</a>.</p>\n": "<h1><a href=\"index.html\">\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9 (AliBi) \u5b9f\u9a13\u306b\u3088\u308b\u6ce8\u610f</a></h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f\u3001AliBi \u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u306e PyTorch \u5b9f\u9a13\u3067\u3059\u3002</a></p>\n<p><a href=\"../gpt/index.html\">\u3053\u308c\u306f\u5f53\u793e\u306eGPT\u30e2\u30c7\u30eb\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>We extend <a href=\"../gpt/index.html\">GPT configurations</a> and change the attention mechanism.</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p><a href=\"../gpt/index.html\">GPT\u69cb\u6210\u3092\u62e1\u5f35\u3057</a>\u3001\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e1\u30ab\u30cb\u30ba\u30e0\u3092\u5909\u66f4\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h3>ALiBi based Transformer configurations</h3>\n": "<h3>AliBi \u30d9\u30fc\u30b9\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create an ALiBi attention module</p>\n": "<p>AliBi \u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u4f5c\u6210\u3059\u308b</p>\n",
|
||||
"<p> Log losses at the initial and final tokens</p>\n": "<p>\u6700\u521d\u306e\u30c8\u30fc\u30af\u30f3\u3068\u6700\u5f8c\u306e\u30c8\u30fc\u30af\u30f3\u3067\u640d\u5931\u3092\u8a18\u9332\u3059\u308b</p>\n",
|
||||
"<p> Shuffled validation data loader with <span translate=no>_^_0_^_</span> sequence length</p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b9\u9577\u306e\u30b7\u30e3\u30c3\u30d5\u30eb\u691c\u8a3c\u30c7\u30fc\u30bf\u30ed\u30fc\u30c0\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>'text': 'tiny_shakespeare_no_split', </p>\n": "<p>'\u30c6\u30ad\u30b9\u30c8': 'tiny_shakespeare_no_split'\u3001</p>\n",
|
||||
"<p>ALiBi based transformer (defined below) </p>\n": "<p>ALiBi \u30d9\u30fc\u30b9\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc (\u4ee5\u4e0b\u306b\u5b9a\u7fa9)</p>\n",
|
||||
"<p>ALiBi doesn't use positional embeddings </p>\n": "<p>AliBi \u306f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u3092\u4f7f\u7528\u3057\u307e\u305b\u3093</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT \u306f GELU \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u3092\u4f7f\u7528\u3057\u3066\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u3092\u884c\u3044\u307e\u3059</p>\n",
|
||||
"<p>If there are more tokens that the training sequence length (during validation), </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3088\u308a\u3082\u591a\u304f\u306e\u30c8\u30fc\u30af\u30f3\u304c\u3042\u308b\u5834\u5408 (\u691c\u8a3c\u4e2d)\u3001</p>\n",
|
||||
"<p>Log the loss at the final token </p>\n": "<p>\u6700\u5f8c\u306e\u30c8\u30fc\u30af\u30f3\u3067\u640d\u5931\u3092\u8a18\u9332\u3059\u308b</p>\n",
|
||||
"<p>Log the loss at the first token </p>\n": "<p>\u6700\u521d\u306e\u30c8\u30fc\u30af\u30f3\u3067\u640d\u5931\u3092\u8a18\u9332\u3059\u308b</p>\n",
|
||||
"<p>Log the loss at training sequence length </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3067\u640d\u5931\u3092\u8a18\u9332\u3057\u307e\u3059</p>\n",
|
||||
"<p>Longer validation set </p>\n": "<p>\u3088\u308a\u9577\u3044\u691c\u8a3c\u30bb\u30c3\u30c8</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set all attention mechanisms to ALiBi </p>\n": "<p>\u3059\u3079\u3066\u306e\u6ce8\u610f\u30e1\u30ab\u30cb\u30ba\u30e0\u3092AliBi\u306b\u8a2d\u5b9a</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
|
||||
"<p>Transformer configurations </p>\n": "<p>\u5909\u5727\u5668\u69cb\u6210</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"Attention with Linear Biases (ALiBi) Experiment": "\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9 (AliBi) \u5b9f\u9a13\u306b\u3088\u308b\u6ce8\u610f",
|
||||
"This experiment trains an Attention with Linear Biases (ALiBi) based model on Tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001Tiny Shakespeare\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u57fa\u3065\u3044\u3066\u3001\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\uff08AliBi\uff09\u306b\u57fa\u3065\u304f\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Attention with Linear Biases (ALiBi)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">ALiBi model</a>.</p>\n<p>This is based on <a href=\"../gpt/index.html\">our GPT model</a>.</p>\n<p><a href=\"https://app.labml.ai/run/1454f9ba044a11ed8364e5e321a405ac\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 (\u0d85\u0dbd\u0dd2\u0db6\u0dd3) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4</a> \u0d9a\u0dbb\u0db1\u0dca\u0db1</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">\u0d85\u0dbd\u0dd2\u0db6\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p>\u0db8\u0dd9\u0dba <a href=\"../gpt/index.html\">\u0d85\u0db4\u0d9c\u0dda GPT \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</a>\u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda. </p>\n<p><a href=\"https://app.labml.ai/run/1454f9ba044a11ed8364e5e321a405ac\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Configurations</h2>\n<p>We extend <a href=\"../gpt/index.html\">GPT configurations</a> and change the attention mechanism.</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0d85\u0db4\u0dd2 <a href=\"../gpt/index.html\">GPT \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a> \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dbb \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dba\u0dcf\u0db1\u0dca\u0dad\u0dca\u0dbb\u0dab\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
|
||||
"<h3>ALiBi based Transformer configurations</h3>\n": "<h3>\u0d85\u0dbd\u0dd2\u0db6\u0dd3\u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create an ALiBi attention module</p>\n": "<p> \u0d85\u0dbd\u0dd2\u0db6\u0dd3\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p> Log losses at the initial and final tokens</p>\n": "<p> \u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a\u0dc4\u0dcf \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dbd\u0daf\u0dd3 \u0db4\u0dcf\u0da9\u0dd4 \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p> Shuffled validation data loader with <span translate=no>_^_0_^_</span> sequence length</p>\n": "<p> <span translate=no>_^_0_^_</span> \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0daf\u0dd2\u0d9c \u0dc3\u0dc4\u0dd2\u0dad \u0dc0\u0dbd\u0d82\u0d9c\u0dd4\u0d9a\u0dbb\u0dab \u0daf\u0dad\u0dca\u0dad \u0db4\u0dd0\u0da7\u0dc0\u0dd4\u0db8 \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dbb \u0d87\u0dad</p>\n",
|
||||
"<p>'text': 'tiny_shakespeare_no_split', </p>\n": "<p>'text ':' \u0da7\u0dd2\u0db1\u0dd2_\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca_\u0db1\u0ddc_\u0db6\u0dd9\u0daf\u0dd3\u0db8\u0dca ', </p>\n",
|
||||
"<p>ALiBi based transformer (defined below) </p>\n": "<p>\u0d85\u0dbd\u0dd2\u0db6\u0dd3\u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca (\u0db4\u0dc4\u0dad \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad) </p>\n",
|
||||
"<p>ALiBi doesn't use positional embeddings </p>\n": "<p>AliBi\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0dba\u0dd2 </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0d9c\u0dad \u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0ddd\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf GELU \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </p>\n",
|
||||
"<p>If there are more tokens that the training sequence length (during validation), </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda \u0daf\u0dd2\u0d9c (\u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3) \u0dc0\u0dd0\u0da9\u0dd2 \u0da7\u0ddd\u0d9a\u0db1 \u0dad\u0dd2\u0db6\u0dda \u0db1\u0db8\u0dca, </p>\n",
|
||||
"<p>Log the loss at the final token </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the loss at the first token </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the loss at training sequence length </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda \u0daf\u0dd2\u0d9c \u0daf\u0dd3 \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Longer validation set </p>\n": "<p>\u0daf\u0dd2\u0d9c\u0dd4\u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set all attention mechanisms to ALiBi </p>\n": "<p>\u0dc3\u0dd2\u0dba\u0dc5\u0dd4\u0db8\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dba\u0dcf\u0db1\u0dca\u0dad\u0dca\u0dbb\u0dab \u0d85\u0dbd\u0dd2\u0db6\u0dd3 \u0dc0\u0dd9\u0dad \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Transformer configurations </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"Attention with Linear Biases (ALiBi) Experiment": "\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 (\u0d85\u0dbd\u0dd2\u0db6\u0dd3) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1",
|
||||
"This experiment trains an Attention with Linear Biases (ALiBi) based model on Tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d85\u0d9c\u0dad\u0dd3\u0db1\u0dca (\u0d85\u0dbd\u0dd2\u0db6\u0dd3) \u0dc3\u0db8\u0d9f \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Attention with Linear Biases (ALiBi)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">ALiBi model</a>.</p>\n<p>This is based on <a href=\"../gpt/index.html\">our GPT model</a>.</p>\n": "<h1><a href=\"index.html\">\u7ebf\u6027\u504f\u5dee\uff08AliBI\uff09\u5b9e\u9a8c\u4e2d\u7684\u6ce8\u610f\u529b</a></h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3 A <a href=\"index.html\">liBI \u6a21\u578b</a>\u3002</p>\n<p>\u8fd9\u662f\u57fa\u4e8e<a href=\"../gpt/index.html\">\u6211\u4eec\u7684 GPT \u6a21\u578b</a>\u3002</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>We extend <a href=\"../gpt/index.html\">GPT configurations</a> and change the attention mechanism.</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u6211\u4eec\u6269\u5c55<a href=\"../gpt/index.html\">\u4e86 GPT \u914d\u7f6e</a>\u5e76\u66f4\u6539\u4e86\u6ce8\u610f\u673a\u5236\u3002</p>\n",
|
||||
"<h3>ALiBi based Transformer configurations</h3>\n": "<h3>\u57fa\u4e8e AliBI \u7684\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create an ALiBi attention module</p>\n": "<p>\u521b\u5efa\u4e00\u4e2a AliBI \u6ce8\u610f\u529b\u6a21\u5757</p>\n",
|
||||
"<p> Log losses at the initial and final tokens</p>\n": "<p>\u8bb0\u5f55\u521d\u59cb\u548c\u6700\u7ec8\u4ee3\u5e01\u7684\u635f\u5931</p>\n",
|
||||
"<p> Shuffled validation data loader with <span translate=no>_^_0_^_</span> sequence length</p>\n": "<p>\u4f7f\u7528<span translate=no>_^_0_^_</span>\u5e8f\u5217\u957f\u5ea6\u6539\u7ec4\u9a8c\u8bc1\u6570\u636e\u52a0\u8f7d\u5668</p>\n",
|
||||
"<p>'text': 'tiny_shakespeare_no_split', </p>\n": "<p>'text': 'tiny_shakespeare_no_split '\uff0c</p>\n",
|
||||
"<p>ALiBi based transformer (defined below) </p>\n": "<p>\u57fa\u4e8e AliBI \u7684\u8f6c\u6362\u5668\uff08\u5b9a\u4e49\u89c1\u4e0b\u6587\uff09</p>\n",
|
||||
"<p>ALiBi doesn't use positional embeddings </p>\n": "<p>AliBI \u4e0d\u4f7f\u7528\u4f4d\u7f6e\u5d4c\u5165</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT \u4f7f\u7528 GELU \u6fc0\u6d3b\u8fdb\u884c\u4f4d\u7f6e\u660e\u667a\u524d\u9988</p>\n",
|
||||
"<p>If there are more tokens that the training sequence length (during validation), </p>\n": "<p>\u5982\u679c\u8bad\u7ec3\u5e8f\u5217\u957f\u5ea6\uff08\u5728\u9a8c\u8bc1\u671f\u95f4\uff09\u6709\u66f4\u591a\u7684\u4ee4\u724c\uff0c</p>\n",
|
||||
"<p>Log the loss at the final token </p>\n": "<p>\u8bb0\u5f55\u6700\u7ec8\u4ee3\u5e01\u7684\u635f\u5931</p>\n",
|
||||
"<p>Log the loss at the first token </p>\n": "<p>\u8bb0\u5f55\u7b2c\u4e00\u4e2a\u4ee4\u724c\u7684\u635f\u5931</p>\n",
|
||||
"<p>Log the loss at training sequence length </p>\n": "<p>\u8bb0\u5f55\u8bad\u7ec3\u5e8f\u5217\u957f\u5ea6\u7684\u635f\u5931</p>\n",
|
||||
"<p>Longer validation set </p>\n": "<p>\u66f4\u957f\u7684\u9a8c\u8bc1\u96c6</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set all attention mechanisms to ALiBi </p>\n": "<p>\u5c06\u6240\u6709\u5173\u6ce8\u673a\u5236\u8bbe\u7f6e\u4e3a AliBI</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
|
||||
"<p>Transformer configurations </p>\n": "<p>\u53d8\u538b\u5668\u914d\u7f6e</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"Attention with Linear Biases (ALiBi) Experiment": "\u6ce8\u610f\u7ebf\u6027\u504f\u5dee (AliBi) \u5b9e\u9a8c",
|
||||
"This experiment trains an Attention with Linear Biases (ALiBi) based model on Tiny Shakespeare dataset.": "\u672c\u5b9e\u9a8c\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u57fa\u4e8e\u7ebf\u6027\u504f\u5dee\u6ce8\u610f\u529b\uff08AliBi\uff09\u7684\u6a21\u578b\u3002"
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1>Transformer Auto-Regression Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset).</p>\n": "<h1>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u52d5\u56de\u5e30\u5b9f\u9a13</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/1706.03762\">\u5fc5\u8981\u306a\u306e\u306f\u6ce8\u610f\u3060\u3051</a>\u300d\u3067\u7d39\u4ecb\u3057\u305f\u30b7\u30f3\u30d7\u30eb\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092NLP\u81ea\u52d5\u56de\u5e30\u30bf\u30b9\u30af\uff08Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u7528\uff09\u3067\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n": "<h2>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u4ee5\u4e0b\u304b\u3089\u7d99\u627f\u3055\u308c\u307e\u3059 <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create GPT model and initialize weights</p>\n": "<p>GPT \u30e2\u30c7\u30eb\u306e\u4f5c\u6210\u3068\u91cd\u307f\u306e\u521d\u671f\u5316</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u30de\u30b9\u30af\u304c\u521d\u671f\u5316\u3055\u308c\u3066\u3044\u306a\u3044\u5834\u5408\u3084\u30de\u30b9\u30af\u306e\u30b5\u30a4\u30ba\u304c\u7570\u306a\u308b\u5834\u5408\u306f\u3001\u5f8c\u7d9a\u306e\u30de\u30b9\u30af\u3092\u4f5c\u6210\u3057\u307e\u3059</p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT \u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3057\u307e\u3059\uff08\u30c8\u30ec\u30fc\u30ca\u30fc\u306fRNN\u3067\u3082\u4f7f\u7528\u3055\u308c\u308b\u305f\u3081\u30012\u756a\u76ee\u306e\u5024\u306f\u72b6\u614b\u7528\u3067\u3059\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u6b21\u306b\u30de\u30b9\u30af\u3059\u308b\u3068\u3001\u30c8\u30fc\u30af\u30f3\u304c\u30de\u30b9\u30af\u3055\u308c\u3001\u5c06\u6765\u306e\u30c8\u30fc\u30af\u30f3\u304c\u898b\u3048\u306a\u304f\u306a\u308a\u307e\u3059</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u30de\u30b9\u30af\u306f\u6700\u521d\u306e\u547c\u3073\u51fa\u3057\u3067\u521d\u671f\u5316\u3055\u308c\u307e\u3059</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32 \u30a8\u30dd\u30c3\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../models.html#Encoder\">\u5909\u5727\u5668\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span><a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u306f\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3059 (\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u4ed8\u304d)</a></li>\n</ul><li><span translate=no>_^_2_^_</span><a href=\"../models.html#Generator\">\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u6700\u5f8c\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u3067\u3059</a>\u3002</li>\n",
|
||||
"This trains a simple transformer model on NLP auto-regression.": "\u3053\u308c\u306b\u3088\u308a\u3001\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u306b NLP \u81ea\u52d5\u56de\u5e30\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u884c\u3044\u307e\u3059\u3002",
|
||||
"Transformer Auto-Regression Experiment": "\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u52d5\u56de\u5e30\u5b9f\u9a13"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1>Transformer Auto-Regression Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset).</p>\n": "<h1>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u0db8\u0dd9\u0dba \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd3 \u0d87\u0dad\u0dd2 \u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2 <a href=\"https://arxiv.org/abs/1706.03762\">\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba-\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0d94\u0db6\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc3\u0dd2\u0dba\u0dbd\u0dca\u0dbd</a> (\u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0dc3\u0db8\u0d9f).</p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0db1\u0dca\u0db1\u0dda <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create GPT model and initialize weights</p>\n": "<p> GPT\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 \u0dc3\u0dc4 \u0db6\u0dbb \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db1\u0db8\u0dca \u0dc4\u0ddd \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd9\u0dc4\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0db1\u0db8\u0dca \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dda\u0dad\u0db1 \u0d9a\u0dca\u0dbb\u0db8 \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0dcf\u0db7\u0db4\u0dca\u0dbb\u0dad\u0dd2 results \u0dbd (\u0daf\u0dd9\u0dc0\u0db1 \u0d85\u0d9c\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dda, \u0db8\u0db1\u0dca\u0daf \u0d85\u0db4\u0d9c\u0dda \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4 RNs \u0dc3\u0db8\u0d9f \u0daf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2) </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab, \u0d85\u0db1\u0dcf\u0d9c\u0dad \u0da7\u0ddd\u0d9a\u0db1 \u0daf\u0dd0\u0d9a\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dc3\u0d82 \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0d87\u0db8\u0dad\u0dd4\u0db8\u0dd9\u0db1\u0dca \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad </p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32\u0dc0\u0dba\u0dc3 \u0d85\u0dc0\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"../models.html#Encoder\">\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a> </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0da7\u0ddd\u0d9a\u0db1\u0dca <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba (\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab \u0dc3\u0db8\u0d9f)</a> </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1 <a href=\"../models.html#Generator\">\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dba\u0dd2</a> . </li></ul>\n",
|
||||
"This trains a simple transformer model on NLP auto-regression.": "\u0db8\u0dd9\u0dba \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3\u0dad\u0dca\u0dc0\u0dba \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2.",
|
||||
"Transformer Auto-Regression Experiment": "\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"<h1>Transformer Auto-Regression Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset).</p>\n": "<h1>\u53d8\u538b\u5668\u81ea\u52a8\u56de\u5f52\u5b9e\u9a8c</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n<p>\u8fd9\u5c06\u8bad\u7ec3\u4e00\u4e2a\u5728 NLP \u81ea\u52a8\u56de\u5f52\u4efb\u52a1\uff08\u4f7f\u7528 Tiny Shakespeare \u6570\u636e\u96c6\uff09\u4e2d\u5f15\u5165\u7684 \u201c<a href=\"https://arxiv.org/abs/1706.03762\">\u6ce8\u610f\u529b\u5c31\u662f\u4f60\u6240\u9700\u8981</a>\u7684\u201d \u7b80\u5355\u53d8\u538b\u5668\u3002</p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n": "<h2>\u81ea\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create GPT model and initialize weights</p>\n": "<p>\u521b\u5efa GPT \u6a21\u578b\u5e76\u521d\u59cb\u5316\u6743\u91cd</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u5982\u679c\u63a9\u7801\u672a\u521d\u59cb\u5316\u6216\u63a9\u7801\u5927\u5c0f\u4e0d\u540c\uff0c\u5219\u521b\u5efa\u540e\u7eed\u63a9\u7801</p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT \u578b\u53f7</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u83b7\u53d6\u65e5\u5fd7</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f7f\u7528\u4f4d\u7f6e\u7f16\u7801\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u578b\u53f7\u5c3a\u5bf8</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c\uff08\u7b2c\u4e8c\u4e2a\u503c\u7528\u4e8e\u72b6\u6001\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u8bad\u7ec3\u5668\u4e5f\u4e0e RNN \u4e00\u8d77\u4f7f\u7528\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u540e\u7eed\u7684\u63a9\u7801\uff0c\u5c06\u63a9\u76d6\u4ee4\u724c\u4ee5\u514d\u770b\u5230\u672a\u6765\u7684\u4ee3\u5e01</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u63a9\u7801\u5c06\u5728\u7b2c\u4e00\u6b21\u8c03\u7528\u65f6\u521d\u59cb\u5316</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>\u8bad\u7ec3 32 \u4e2a\u65f6\u4ee3</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668<a href=\"../models.html#Encoder\">\u7f16\u7801\u5668</a></li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c<a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u5d4c\u5165\u6a21\u5757\uff08\u5e26\u6709\u4f4d\u7f6e\u7f16\u7801\uff09</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u7ed9<a href=\"../models.html#Generator\">\u51fa logit \u7684\u6700\u540e\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42</a>\u3002</li></ul>\n",
|
||||
"This trains a simple transformer model on NLP auto-regression.": "\u8fd9\u4f1a\u5728 NLP \u81ea\u52a8\u56de\u5f52\u4e0a\u8bad\u7ec3\u4e00\u4e2a\u7b80\u5355\u7684\u53d8\u538b\u5668\u6a21\u578b\u3002",
|
||||
"Transformer Auto-Regression Experiment": "\u53d8\u538b\u5668\u81ea\u52a8\u56de\u5f52\u5b9e\u9a8c"
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"<h1>Transformer Auto-Regression Experiment with <a href=\"../../optimizers/sophia.html\">Sophia-G optimizer</a></h1>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset) with <a href=\"../../optimizers/sophia.html\">Sophia-G optimizer</a>.</p>\n": "<h1>Transformer Auto-Regression Experiment with <a href=\"../../optimizers/sophia.html\">Sophia-G optimizer</a></h1>\n<p>This trains a simple transformer introduced in <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a> on an NLP auto-regression task (with Tiny Shakespeare dataset) with <a href=\"../../optimizers/sophia.html\">Sophia-G optimizer</a>.</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"autoregressive_experiment.html\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>Configurations</h2>\n<p>This inherits from <a href=\"autoregressive_experiment.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Training or validation step with Gauss-Newton-Bartlett (GNB) Hessian diagonal estimator</h3>\n": "<h3>Training or validation step with Gauss-Newton-Bartlett (GNB) Hessian diagonal estimator</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>Batch size <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>Calculate and log accuracy </p>\n",
|
||||
"<p>Calculate and log loss </p>\n": "<p>Calculate and log loss </p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>Calculate gradients </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>Clear the gradients </p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>Clip gradients </p>\n",
|
||||
"<p>Create a categorical distribution from logits </p>\n": "<p>Create a categorical distribution from logits </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>Create configs </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>Create experiment </p>\n",
|
||||
"<p>Estimate the Hessian diagonal every <span translate=no>_^_0_^_</span> steps </p>\n": "<p>Estimate the Hessian diagonal every <span translate=no>_^_0_^_</span> steps </p>\n",
|
||||
"<p>Get model outputs </p>\n": "<p>Get model outputs </p>\n",
|
||||
"<p>Get model outputs. It's returning a tuple for states when using RNNs. This is not implemented yet. \ud83d\ude1c </p>\n": "<p>Get model outputs. It's returning a tuple for states when using RNNs. This is not implemented yet. \ud83d\ude1c </p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>Log the model parameters and gradients on last batch of every epoch </p>\n",
|
||||
"<p>Model size </p>\n": "<p>Model size </p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>Move data to the device </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>Override configurations </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>Prompt separator is blank </p>\n",
|
||||
"<p>Run training </p>\n": "<p>Run training </p>\n",
|
||||
"<p>Sample <span translate=no>_^_0_^_</span> </p>\n": "<p>Sample <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>Save the tracked metrics </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>Set models for saving and loading </p>\n",
|
||||
"<p>Set training/eval mode </p>\n": "<p>Set training/eval mode </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>Start the experiment </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>Starting prompt for sampling </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>Take optimizer step </p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>Train for 32 epochs </p>\n",
|
||||
"<p>Train the model </p>\n": "<p>Train the model </p>\n",
|
||||
"<p>Update EMA Hessian diagonal</p>\n<span translate=no>_^_0_^_</span><p> </p>\n": "<p>Update EMA Hessian diagonal</p>\n<span translate=no>_^_0_^_</span><p> </p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>Update global step (number of tokens processed) when in training mode </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/sophia.html\">Sophia optimizer</a> </p>\n": "<p>Use <a href=\"../../optimizers/sophia.html\">Sophia optimizer</a> </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>Use Tiny Shakespeare dataset </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>Use character level tokenizer </p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>Whether to capture model outputs </p>\n",
|
||||
"This trains a simple transformer model on NLP auto-regression with Sophia-G optimizer.": "This trains a simple transformer model on NLP auto-regression with Sophia-G optimizer.",
|
||||
"Transformer Auto-Regression Experiment with [Sophia-G optimizer](../../optimizers/sophia.html)": "Transformer Auto-Regression Experiment with [Sophia-G optimizer](../../optimizers/sophia.html)"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"<h1>Compressive Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a compressive transformer model.</p>\n": "<h1>\u5727\u7e2e\u5909\u5727\u5668\u5b9f\u9a13</h1>\n<p>\u3053\u308c\u306f\u3001\u5727\u7e2e\u30c8\u30e9\u30f3\u30b9\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u306e PyTorch \u5b9f\u9a13\u3067\u3059\u3002</p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52d5\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configurations can and will be overridden when we start the experiment.</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u69cb\u6210\u306f\u3001\u5b9f\u9a13\u3092\u958b\u59cb\u3059\u308b\u3068\u304d\u306b\u4e0a\u66f8\u304d\u3067\u304d\u307e\u3059\u3002\u307e\u305f\u3001\u4eca\u5f8c\u5909\u66f4\u3059\u308b\u4e88\u5b9a\u3067\u3059\u3002</p>\n",
|
||||
"<h3>Initialize the attention reconstruction loss</h3>\n": "<h3>\u6ce8\u610f\u529b\u518d\u69cb\u7bc9\u30ed\u30b9\u3092\u521d\u671f\u5316</h3>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u3092\u521d\u671f\u5316</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u5b9f\u9a13\u3092\u5b9f\u884c\u3059\u308b</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\u5b9a\u671f\u7684\u306b\u30b5\u30f3\u30d7\u30eb\u3092\u751f\u6210\u3059\u308b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u6a5f\u80fd</h3>\n",
|
||||
"<h3>Training/validation step</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0/\u691c\u8a3c\u30b9\u30c6\u30c3\u30d7</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Concatenate new memories and compress the oldest memories.</p>\n": "<p>\u65b0\u3057\u3044\u8a18\u61b6\u3092\u9023\u7d50\u3057\u3001\u6700\u3082\u53e4\u3044\u8a18\u61b6\u3092\u5727\u7e2e\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u3059\u308b\u8a2d\u5b9a\u306e\u8f9e\u66f8</p>\n",
|
||||
"<p>A list to keep memories that need to be compressed for each layer. </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u3054\u3068\u306b\u5727\u7e2e\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b\u30e1\u30e2\u30ea\u3092\u4fdd\u5b58\u3059\u308b\u305f\u3081\u306e\u30ea\u30b9\u30c8\u3002</p>\n",
|
||||
"<p>A list to keep the memories that do not get compressed for each layer. </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u3054\u3068\u306b\u5727\u7e2e\u3055\u308c\u306a\u3044\u30e1\u30e2\u30ea\u3092\u4fdd\u5b58\u3059\u308b\u305f\u3081\u306e\u30ea\u30b9\u30c8\u3002</p>\n",
|
||||
"<p>Add a hook to log module outputs </p>\n": "<p>\u30e2\u30b8\u30e5\u30fc\u30eb\u51fa\u529b\u3092\u30ed\u30b0\u306b\u8a18\u9332\u3059\u308b\u30d5\u30c3\u30af\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Add attention reconstruction loss to loss </p>\n": "<p>\u640d\u5931\u306b\u6ce8\u610f\u518d\u69cb\u7bc9\u640d\u5931\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u30ed\u30ae\u30f3\u30b0\u7528\u306e\u4e88\u6e2c\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u4e88\u6e2c\u3092\u30d7\u30ed\u30f3\u30d7\u30c8\u306b\u8ffd\u52a0</p>\n",
|
||||
"<p>Attention Reconstruction Loss </p>\n": "<p>\u6ce8\u610f\u529b\u518d\u5efa\u30ed\u30b9</p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u7cbe\u5ea6\u306e\u8a08\u7b97\u3068\u8a18\u9332</p>\n",
|
||||
"<p>Calculate and log cross entropy loss </p>\n": "<p>\u30af\u30ed\u30b9\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u640d\u5931\u306e\u8a08\u7b97\u3068\u8a18\u9332</p>\n",
|
||||
"<p>Calculate attention reconstruction loss if memories were compressed in this step </p>\n": "<p>\u3053\u306e\u30b9\u30c6\u30c3\u30d7\u3067\u8a18\u61b6\u304c\u5727\u7e2e\u3055\u308c\u305f\u5834\u5408\u306e\u6ce8\u610f\u518d\u69cb\u6210\u640d\u5931\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u52fe\u914d\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate the number of compressed memories to make <span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is the number of memories we have and <span translate=no>_^_2_^_</span> is the maximum number of memories we maintain (<span translate=no>_^_3_^_</span>). </p>\n": "<p>\u4f5c\u6210\u3059\u308b\u5727\u7e2e\u30e1\u30e2\u30ea\u306e\u6570\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002\u3053\u3053\u3067<span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u306f\u4fdd\u6301\u3059\u308b\u30e1\u30e2\u30ea\u306e\u6700\u5927\u6570\u3001<span translate=no>_^_2_^_</span>\u306f\u4fdd\u6301\u3059\u308b\u30e1\u30e2\u30ea\u306e\u6700\u5927\u6570 (<span translate=no>_^_3_^_</span>)\u3002</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30af\u30ea\u30a2</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u30af\u30ea\u30c3\u30d7\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Collect memories to compress </p>\n": "<p>\u601d\u3044\u51fa\u3092\u96c6\u3081\u3066\u5727\u7e2e</p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u5370\u5237\u7528\u306e\u51fa\u529b\u3092\u53ce\u96c6</p>\n",
|
||||
"<p>Collect remaining memories </p>\n": "<p>\u6b8b\u308a\u306e\u601d\u3044\u51fa\u3092\u96c6\u3081\u3088\u3046</p>\n",
|
||||
"<p>Compress the memories </p>\n": "<p>\u601d\u3044\u51fa\u3092\u5727\u7e2e</p>\n",
|
||||
"<p>Compress the oldest memories if there are more memories than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3088\u308a\u591a\u304f\u306e\u30e1\u30e2\u30ea\u304c\u3042\u308b\u5834\u5408\u306f\u3001\u6700\u3082\u53e4\u3044\u30e1\u30e2\u30ea\u3092\u5727\u7e2e\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Compressed memory length </p>\n": "<p>\u5727\u7e2e\u30e1\u30e2\u30ea\u9577</p>\n",
|
||||
"<p>Compression rate </p>\n": "<p>\u5727\u7e2e\u7387</p>\n",
|
||||
"<p>Concatenate new memories with old memory </p>\n": "<p>\u65b0\u3057\u3044\u8a18\u61b6\u3068\u53e4\u3044\u8a18\u61b6\u3092\u3064\u306a\u3052\u308b</p>\n",
|
||||
"<p>Concatenate newly compressed memories with old compressed memories </p>\n": "<p>\u65b0\u3057\u304f\u5727\u7e2e\u3055\u308c\u305f\u30e1\u30e2\u30ea\u3092\u53e4\u3044\u5727\u7e2e\u30e1\u30e2\u30ea\u3068\u9023\u7d50\u3059\u308b</p>\n",
|
||||
"<p>Concatenate the masks if there is memory </p>\n": "<p>\u30e1\u30e2\u30ea\u304c\u3042\u308b\u5834\u5408\u306f\u30de\u30b9\u30af\u3092\u9023\u7d50\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Create a subsequent mask for tokens </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u30de\u30b9\u30af\u3092\u5f8c\u304b\u3089\u4f5c\u6210</p>\n",
|
||||
"<p>Create an all ones (full visibility) mask for memory </p>\n": "<p>\u30e1\u30e2\u30ea\u7528\u306e\u30aa\u30fc\u30eb\u30ef\u30f3 (\u30d5\u30eb\u30d3\u30b8\u30d3\u30ea\u30c6\u30a3) \u30de\u30b9\u30af\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Do not print the attention reconstruction loss in the terminal </p>\n": "<p>\u7aef\u672b\u306b\u6ce8\u610f\u518d\u69cb\u6210\u30ed\u30b9\u3092\u5370\u5237\u3057\u306a\u3044\u3067\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8131\u843d\u78ba\u7387</p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u6700\u7d42\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210</p>\n",
|
||||
"<p>Get attention reconstruction loss </p>\n": "<p>\u6ce8\u610f\u3092\u5411\u3051\u3066\u518d\u5efa\u30ed\u30b9</p>\n",
|
||||
"<p>Get memories </p>\n": "<p>\u601d\u3044\u51fa\u3092\u30b2\u30c3\u30c8</p>\n",
|
||||
"<p>Get memory and compressed memory </p>\n": "<p>\u30e1\u30e2\u30ea\u3068\u5727\u7e2e\u30e1\u30e2\u30ea\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u30e2\u30c7\u30eb\u4e88\u6e2c\u3092\u53d6\u5f97 (\u6b32\u5f35\u308a)</p>\n",
|
||||
"<p>If the configurations specify not to use memory </p>\n": "<p>\u69cb\u6210\u3067\u30e1\u30e2\u30ea\u3092\u4f7f\u7528\u3057\u306a\u3044\u3088\u3046\u6307\u5b9a\u3055\u308c\u3066\u3044\u308b\u5834\u5408</p>\n",
|
||||
"<p>If there are no old compressed memories </p>\n": "<p>\u53e4\u3044\u5727\u7e2e\u30e1\u30e2\u30ea\u304c\u306a\u3044\u5834\u5408</p>\n",
|
||||
"<p>Iterate through memories of each layer. </p>\n": "<p>\u5404\u30ec\u30a4\u30e4\u30fc\u306e\u30e1\u30e2\u30ea\u3092\u7e70\u308a\u8fd4\u3057\u51e6\u7406\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u5404\u30a8\u30dd\u30c3\u30af\u306e\u6700\u5f8c\u306e\u30d0\u30c3\u30c1\u3067\u30e2\u30c7\u30eb\u30d1\u30e9\u30e1\u30fc\u30bf\u3068\u52fe\u914d\u3092\u8a18\u9332\u3057\u307e\u3059</p>\n",
|
||||
"<p>Masks </p>\n": "<p>\u30de\u30b9\u30af</p>\n",
|
||||
"<p>Merge and compress memory </p>\n": "<p>\u30e1\u30e2\u30ea\u306e\u7d71\u5408\u3068\u5727\u7e2e</p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u30c7\u30fc\u30bf\u3092\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Move to device </p>\n": "<p>\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>No memories are compressed if the number of memories is less than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30e1\u30e2\u30ea\u306e\u6570\u304c\u4ee5\u4e0b\u306e\u5834\u5408\u3001\u30e1\u30e2\u30ea\u306f\u5727\u7e2e\u3055\u308c\u307e\u305b\u3093 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u6570</p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN \u96a0\u308c\u30ec\u30a4\u30e4\u30fc\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u6570</p>\n",
|
||||
"<p>Number of memories to compress <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5727\u7e2e\u3059\u308b\u30e1\u30e2\u30ea\u306e\u6570 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of memories to keep </p>\n": "<p>\u4fdd\u5b58\u3059\u308b\u30e1\u30e2\u30ea\u306e\u6570</p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u5909\u5727\u5668\u5c64\u306e\u6570</p>\n",
|
||||
"<p>Only feed the last character to model in next iteration, rest will go in as memories </p>\n": "<p>\u6b21\u306e\u30a4\u30c6\u30ec\u30fc\u30b7\u30e7\u30f3\u3067\u306f\u6700\u5f8c\u306e\u6587\u5b57\u3060\u3051\u3092\u30e2\u30c7\u30eb\u306b\u30d5\u30a3\u30fc\u30c9\u3057\u3001\u6b8b\u308a\u306f\u30e1\u30e2\u30ea\u3068\u3057\u3066\u6b8b\u308a\u307e\u3059</p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u51fa\u529b\u3092\u5370\u5237\u3059\u308b</p>\n",
|
||||
"<p>Return memories and the memories that were compressed. Memories that were compressed are needed for the reconstruction loss computation. </p>\n": "<p>\u30e1\u30e2\u30ea\u3068\u5727\u7e2e\u3055\u308c\u305f\u30e1\u30e2\u30ea\u3092\u8fd4\u3057\u307e\u3059\u3002\u518d\u69cb\u6210\u640d\u5931\u306e\u8a08\u7b97\u306b\u306f\u3001\u5727\u7e2e\u3055\u308c\u305f\u30e1\u30e2\u30ea\u304c\u5fc5\u8981\u3067\u3059</p>\u3002\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u5909\u5727\u5668\u306b\u901a\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Run the model </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>25\u30c8\u30fc\u30af\u30f3\u306e\u30b5\u30f3\u30d7\u30eb</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u8ffd\u8de1\u3057\u305f\u30e1\u30c8\u30ea\u30af\u30b9\u3092\u4fdd\u5b58\u3059\u308b</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u30c8\u30e9\u30c3\u30ab\u30fc\u69cb\u6210\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Split the memories at <span translate=no>_^_0_^_</span> </p>\n": "<p>\u601d\u3044\u51fa\u3092\u5206\u3051\u3066 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u8d77\u52d5\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>State module to maintain memories when switching between training and validation </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b\u3068\u304d\u306b\u30e1\u30e2\u30ea\u3092\u7dad\u6301\u3059\u308b\u30b9\u30c6\u30fc\u30c8\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u6700\u9069\u5316\u306e\u4e00\u6b69\u3092\u8e0f\u307f\u51fa\u3059</p>\n",
|
||||
"<p>This will keep the accuracy metric stats and memories separate for training and validation. </p>\n": "<p>\u3053\u308c\u306b\u3088\u308a\u3001\u7cbe\u5ea6\u30e1\u30c8\u30ea\u30c3\u30af\u306e\u7d71\u8a08\u60c5\u5831\u3068\u30e1\u30e2\u30ea\u304c\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u7528\u306b\u5225\u3005\u306b\u4fdd\u6301\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u3092\u30c8\u30fc\u30af\u30f3\u5316</p>\n",
|
||||
"<p>Total length of the memory and compressed memory (for masks) </p>\n": "<p>\u30e1\u30e2\u30ea\u3068\u5727\u7e2e\u30e1\u30e2\u30ea\u306e\u5408\u8a08\u9577 (\u30de\u30b9\u30af\u7528)</p>\n",
|
||||
"<p>Track attention reconstruction loss </p>\n": "<p>\u30c8\u30e9\u30c3\u30af\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30ea\u30b3\u30f3\u30b9\u30c8\u30e9\u30af\u30b7\u30e7\u30f3\u30fb\u30ed\u30b9</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Truncate old memories </p>\n": "<p>\u53e4\u3044\u601d\u3044\u51fa\u3092\u5207\u308a\u6368\u3066\u308b</p>\n",
|
||||
"<p>Update and compress memory </p>\n": "<p>\u30e1\u30e2\u30ea\u306e\u66f4\u65b0\u3068\u5727\u7e2e</p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30e2\u30fc\u30c9\u6642\u306b\u30b0\u30ed\u30fc\u30d0\u30eb\u30b9\u30c6\u30c3\u30d7 (\u51e6\u7406\u3055\u308c\u305f\u30c8\u30fc\u30af\u30f3\u306e\u6570) \u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Update memories </p>\n": "<p>\u30e1\u30e2\u30ea\u30fc\u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Update the memories </p>\n": "<p>\u601d\u3044\u51fa\u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Use only the subsequent mask otherwise </p>\n": "<p>\u305d\u308c\u4ee5\u5916\u306e\u5834\u5408\u306f\u3001\u5f8c\u7d9a\u306e\u30de\u30b9\u30af\u306e\u307f\u3092\u4f7f\u7528\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u30ad\u30e3\u30d7\u30c1\u30e3\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>memory </p>\n": "<p>\u8a18\u61b6</p>\n",
|
||||
"Compressive Transformer Experiment": "\u5727\u7e2e\u5909\u5727\u5668\u5b9f\u9a13",
|
||||
"This experiment trains a compressive transformer model on tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u5c0f\u3055\u306a\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067\u5727\u7e2e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"<h1>Compressive Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a compressive transformer model.</p>\n": "<h1>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configurations can and will be overridden when we start the experiment.</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0d85\u0db4 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1 \u0dc0\u0dd2\u0da7 \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0d89\u0d9a\u0dca\u0db8\u0dc0\u0dcf \u0dba\u0dcf \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. </p>\n",
|
||||
"<h3>Initialize the attention reconstruction loss</h3>\n": "<h3>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0dc0\u0dbb\u0dd2\u0db1\u0dca \u0dc0\u0dbb \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba</h3>\n",
|
||||
"<h3>Training/validation step</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0/\u0dc0\u0dbd\u0d82\u0d9c\u0dd4\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dd2\u0dba\u0dc0\u0dbb</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Concatenate new memories and compress the oldest memories.</p>\n": "<p> \u0db1\u0dc0\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb \u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dad\u0db8 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf\u0dba\u0dcf\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc1\u0db6\u0dca\u0daf\u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dca </p>\n",
|
||||
"<p>A list to keep memories that need to be compressed for each layer. </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dad\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0dca. </p>\n",
|
||||
"<p>A list to keep the memories that do not get compressed for each layer. </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db1\u0ddc\u0dc0\u0db1 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dad\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0dca. </p>\n",
|
||||
"<p>Add a hook to log module outputs </p>\n": "<p>\u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d9a\u0ddc\u0d9a\u0dca\u0d9a\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add attention reconstruction loss to loss </p>\n": "<p>\u0d85\u0dbd\u0dcf\u0db7\u0dba\u0da7\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u0dbd\u0ddc\u0d9c\u0dca\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0da7\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Attention Reconstruction Loss </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab \u0d85\u0dbd\u0dcf\u0db7\u0dba </p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba </p>\n",
|
||||
"<p>Calculate and log cross entropy loss </p>\n": "<p>\u0dc4\u0dbb\u0dc3\u0dca\u0d91\u0db1\u0dca\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd2 \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate attention reconstruction loss if memories were compressed in this step </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0db8\u0dd9\u0db8 \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db1\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0d9c\u0dab\u0db1\u0dba </p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate the number of compressed memories to make <span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is the number of memories we have and <span translate=no>_^_2_^_</span> is the maximum number of memories we maintain (<span translate=no>_^_3_^_</span>). </p>\n": "<p>\u0dc3\u0dd1\u0daf\u0dd3\u0db8\u0da7\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span>, \u0d85\u0db4 \u0dc3\u0dad\u0dd4\u0dc0 \u0d87\u0dad\u0dd2 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf \u0dc3\u0dc4 <span translate=no>_^_2_^_</span> \u0d85\u0db4 \u0db1\u0da9\u0dad\u0dca\u0dad\u0dd4 \u0d9a\u0dbb\u0db1 \u0d8b\u0db4\u0dbb\u0dd2\u0db8 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 ( <span translate=no>_^_3_^_</span>). </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d89\u0dc0\u0dad\u0dca </p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dca\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a </p>\n",
|
||||
"<p>Collect memories to compress </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Collect remaining memories </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Compress the memories </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dc3\u0d82\u0d9a\u0ddd\u0da0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Compress the oldest memories if there are more memories than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc0\u0da9\u0dcf\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dad\u0dd2\u0db6\u0dda \u0db1\u0db8\u0dca \u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dad\u0db8 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Compressed memory length </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad\u0db8\u0dad\u0d9a \u0daf\u0dd2\u0d9c </p>\n",
|
||||
"<p>Compression rate </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba </p>\n",
|
||||
"<p>Concatenate new memories with old memory </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0db8\u0dad\u0d9a\u0dba \u0dc3\u0db8\u0d9f \u0db1\u0dc0 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Concatenate newly compressed memories with old compressed memories </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0d9f \u0d85\u0dbd\u0dd4\u0dad\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Concatenate the masks if there is memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0d9a\u0dca\u0dad\u0dd2\u0db6\u0dda \u0db1\u0db8\u0dca \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create a subsequent mask for tokens </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create an all ones (full visibility) mask for memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dd2\u0dba\u0dbd\u0dd4 (\u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab \u0daf\u0dd8\u0dc1\u0dca\u0dba\u0dad\u0dcf\u0dc0) \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Do not print the attention reconstruction loss in the terminal </p>\n": "<p>\u0db4\u0dbb\u0dca\u0dba\u0db1\u0dca\u0dad\u0dba\u0dda\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0db1\u0ddc\u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u0d85\u0dad\u0dc4\u0dd0\u0dbb\u0daf\u0dd0\u0db8\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0db1\u0dca\u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get attention reconstruction loss </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get memories </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get memory and compressed memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0dc3\u0dc4 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 (\u0d9a\u0dd1\u0daf\u0dbb) </p>\n",
|
||||
"<p>If the configurations specify not to use memory </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db8\u0dad\u0d9a\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0db1\u0ddc\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db1\u0dd2\u0dba\u0db8 \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dda \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>If there are no old compressed memories </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>Iterate through memories of each layer. </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc4\u0dbb\u0dc4\u0dcf \u0d9c\u0db8\u0db1\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u0dc3\u0dd1\u0db8\u0dba\u0dd4\u0d9c\u0dbd\u0dba\u0d9a\u0db8 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0dc4 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Masks </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 </p>\n",
|
||||
"<p>Merge and compress memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0d92\u0d9a\u0dcf\u0db6\u0daf\u0dca\u0db0 \u0d9a\u0dbb \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dc0\u0dd9\u0dad \u0daf\u0dad\u0dca\u0dad \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move to device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dc0\u0dd9\u0dad \u0d9c\u0dd9\u0db1 \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>No memories are compressed if the number of memories is less than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0d9c\u0dab\u0db1 \u0d85\u0da9\u0dd4 \u0db1\u0db8\u0dca \u0d9a\u0dd2\u0dc3\u0dd2\u0daf\u0dd4 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db1\u0ddc\u0dc0\u0dda <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN\u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of memories to compress <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Number of memories to keep </p>\n": "<p>\u0dad\u0db6\u0dcf\u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc3\u0dca\u0dae\u0dbb \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Only feed the last character to model in next iteration, rest will go in as memories </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0db4\u0dd4\u0db1\u0dbb\u0dcf\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0dda\u0daf\u0dd3 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0da0\u0dbb\u0dd2\u0dad\u0dba \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0da7 \u0db4\u0db8\u0dab\u0d9a\u0dca \u0db4\u0ddd\u0dc2\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1, \u0dc0\u0dd2\u0dc0\u0dda\u0d9a\u0dba \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dbd\u0dd9\u0dc3 \u0d89\u0daf\u0dd2\u0dbb\u0dd2\u0dba\u0da7 \u0dba\u0db1\u0dd4 \u0d87\u0dad </p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Return memories and the memories that were compressed. Memories that were compressed are needed for the reconstruction loss computation. </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dc3\u0dc4 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1\u0dca\u0db1. \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db6\u0dc0 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dda. </p>\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0dc4\u0dbb\u0dc4\u0dcf \u0d91\u0dba \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd25 \u0da7\u0ddd\u0d9a\u0db1 </p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u0dbd\u0dd4\u0dc4\u0dd4\u0db6\u0dd0\u0db3\u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dd0\u0d9a\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Split the memories at <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0daf\u0dd3\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db6\u0dd9\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>State module to maintain memories when switching between training and validation </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4\u0dc0\u0dd3\u0db8\u0dda\u0daf\u0dd3 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db4\u0dc0\u0dad\u0dca\u0dc0\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>This will keep the accuracy metric stats and memories separate for training and validation. </p>\n": "<p>\u0db8\u0dd9\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf \u0db8\u0dd9\u0da7\u0dca\u0dbb\u0dd2\u0d9a\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0db1 \u0dc3\u0dc4 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc0\u0dd9\u0db1\u0db8 \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dd3. </p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca </p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0da7\u0ddd\u0d9a\u0dd9\u0db1\u0dca\u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Total length of the memory and compressed memory (for masks) </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0dda\u0db8\u0dd4\u0dc5\u0dd4 \u0daf\u0dd2\u0d9c \u0dc3\u0dc4 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0dda (\u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dc3\u0db3\u0dc4\u0dcf) </p>\n",
|
||||
"<p>Track attention reconstruction loss </p>\n": "<p>\u0db0\u0dcf\u0dc0\u0db1\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab \u0d85\u0dbd\u0dcf\u0db7\u0dba </p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Truncate old memories </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Update and compress memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db4\u0dca\u0dbb\u0d9a\u0dcf\u0dbb\u0dba\u0dda\u0daf\u0dd3 \u0d9c\u0ddd\u0dbd\u0dd3\u0dba \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 (\u0dc3\u0dd0\u0d9a\u0dc3\u0dd6 \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1) </p>\n",
|
||||
"<p>Update memories </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Update the memories </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use only the subsequent mask otherwise </p>\n": "<p>\u0dc0\u0dd9\u0db1\u0dad\u0dca\u0d86\u0d9a\u0dcf\u0dbb\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d86\u0dc0\u0dbb\u0dab \u0db4\u0db8\u0dab\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d9c\u0dca\u0dbb\u0dc4\u0dab\u0dba \u0d9a\u0dbb \u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba </p>\n",
|
||||
"Compressive Transformer Experiment": "\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains a compressive transformer model on tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db8\u0dad \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"<h1>Compressive Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a compressive transformer model.</p>\n": "<h1>\u538b\u7f29\u53d8\u538b\u5668\u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u4e2a\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3\u538b\u7f29\u53d8\u538b\u5668\u6a21\u578b\u3002</p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52a8\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configurations can and will be overridden when we start the experiment.</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u5f53\u6211\u4eec\u5f00\u59cb\u5b9e\u9a8c\u65f6\uff0c\u9ed8\u8ba4\u914d\u7f6e\u53ef\u4ee5\u800c\u4e14\u5c06\u4f1a\u88ab\u8986\u76d6\u3002</p>\n",
|
||||
"<h3>Initialize the attention reconstruction loss</h3>\n": "<h3>\u521d\u59cb\u5316\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</h3>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u521d\u59cb\u5316\u81ea\u56de\u5f52\u6a21\u578b</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u8fd0\u884c\u5b9e\u9a8c</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u91c7\u6837\u529f\u80fd\u53ef\u5728\u8bad\u7ec3\u65f6\u5b9a\u671f\u751f\u6210\u6837\u672c</h3>\n",
|
||||
"<h3>Training/validation step</h3>\n": "<h3>\u57f9\u8bad/\u9a8c\u8bc1\u6b65\u9aa4</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Concatenate new memories and compress the oldest memories.</p>\n": "<p>\u8fde\u63a5\u65b0\u8bb0\u5fc6\u5e76\u538b\u7f29\u6700\u53e4\u8001\u7684\u8bb0\u5fc6\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u8981\u8986\u76d6\u7684\u914d\u7f6e\u5b57\u5178</p>\n",
|
||||
"<p>A list to keep memories that need to be compressed for each layer. </p>\n": "<p>\u7528\u4e8e\u4fdd\u5b58\u6bcf\u5c42\u9700\u8981\u538b\u7f29\u7684\u5185\u5b58\u7684\u5217\u8868\u3002</p>\n",
|
||||
"<p>A list to keep the memories that do not get compressed for each layer. </p>\n": "<p>\u4e00\u4e2a\u5217\u8868\uff0c\u7528\u4e8e\u4fdd\u5b58\u6bcf\u5c42\u672a\u88ab\u538b\u7f29\u7684\u8bb0\u5fc6\u3002</p>\n",
|
||||
"<p>Add a hook to log module outputs </p>\n": "<p>\u5411\u65e5\u5fd7\u6a21\u5757\u8f93\u51fa\u6dfb\u52a0\u94a9\u5b50</p>\n",
|
||||
"<p>Add attention reconstruction loss to loss </p>\n": "<p>\u5c06\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931\u589e\u52a0\u5230\u635f\u5931</p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u6dfb\u52a0\u65e5\u5fd7\u8bb0\u5f55\u7684\u9884\u6d4b</p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u5c06\u9884\u6d4b\u6dfb\u52a0\u5230\u63d0\u793a\u7b26\u4e2d</p>\n",
|
||||
"<p>Attention Reconstruction Loss </p>\n": "<p>\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u8ba1\u7b97\u548c\u8bb0\u5f55\u7cbe\u5ea6</p>\n",
|
||||
"<p>Calculate and log cross entropy loss </p>\n": "<p>\u8ba1\u7b97\u548c\u8bb0\u5f55\u4ea4\u53c9\u71b5\u635f\u5931</p>\n",
|
||||
"<p>Calculate attention reconstruction loss if memories were compressed in this step </p>\n": "<p>\u5982\u679c\u5728\u6b64\u6b65\u9aa4\u4e2d\u8bb0\u5fc6\u88ab\u538b\u7f29\uff0c\u5219\u8ba1\u7b97\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"<p>Calculate the number of compressed memories to make <span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is the number of memories we have and <span translate=no>_^_2_^_</span> is the maximum number of memories we maintain (<span translate=no>_^_3_^_</span>). </p>\n": "<p>\u8ba1\u7b97\u8981\u5236\u4f5c\u7684\u538b\u7f29\u8bb0\u5fc6\u7684\u6570\u91cf<span translate=no>_^_0_^_</span>\uff0c\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u6211\u4eec\u62e5\u6709\u7684\u8bb0\u5fc6\u6570\u91cf\uff0c<span translate=no>_^_2_^_</span>\u662f\u6211\u4eec\u7ef4\u62a4\u7684\u6700\u5927\u8bb0\u5fc6\u6570\uff08<span translate=no>_^_3_^_</span>)\u3002</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u6e05\u9664\u6e10\u53d8</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u526a\u8f91\u6e10\u53d8</p>\n",
|
||||
"<p>Collect memories to compress </p>\n": "<p>\u6536\u96c6\u8bb0\u5fc6\u8fdb\u884c\u538b\u7f29</p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u6536\u96c6\u8f93\u51fa\u4ee5\u8fdb\u884c\u6253\u5370</p>\n",
|
||||
"<p>Collect remaining memories </p>\n": "<p>\u6536\u96c6\u5269\u4f59\u7684\u8bb0\u5fc6</p>\n",
|
||||
"<p>Compress the memories </p>\n": "<p>\u538b\u7f29\u8bb0\u5fc6</p>\n",
|
||||
"<p>Compress the oldest memories if there are more memories than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u8bb0\u5fc6\u591a\u4e8e\u6700\u65e9\u7684\u8bb0\u5fc6\uff0c\u5219\u538b\u7f29\u6700\u65e9\u7684\u8bb0\u5fc6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Compressed memory length </p>\n": "<p>\u538b\u7f29\u7684\u5185\u5b58\u957f\u5ea6</p>\n",
|
||||
"<p>Compression rate </p>\n": "<p>\u538b\u7f29\u7387</p>\n",
|
||||
"<p>Concatenate new memories with old memory </p>\n": "<p>\u5c06\u65b0\u8bb0\u5fc6\u4e0e\u65e7\u8bb0\u5fc6\u8fde\u63a5\u8d77\u6765</p>\n",
|
||||
"<p>Concatenate newly compressed memories with old compressed memories </p>\n": "<p>\u5c06\u65b0\u538b\u7f29\u7684\u5b58\u50a8\u5668\u4e0e\u65e7\u7684\u538b\u7f29\u5b58\u50a8\u5668\u8fde\u63a5\u8d77\u6765</p>\n",
|
||||
"<p>Concatenate the masks if there is memory </p>\n": "<p>\u5982\u679c\u6709\u5185\u5b58\uff0c\u5219\u8fde\u63a5\u63a9\u7801</p>\n",
|
||||
"<p>Create a subsequent mask for tokens </p>\n": "<p>\u4e3a\u4ee4\u724c\u521b\u5efa\u540e\u7eed\u63a9\u7801</p>\n",
|
||||
"<p>Create an all ones (full visibility) mask for memory </p>\n": "<p>\u4e3a\u5185\u5b58\u521b\u5efa\u4e00\u4e2a\u5168\u4e00\uff08\u5b8c\u5168\u53ef\u89c1\u6027\uff09\u63a9\u7801</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Do not print the attention reconstruction loss in the terminal </p>\n": "<p>\u4e0d\u8981\u5728\u7ec8\u7aef\u4e2d\u6253\u5370\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8f8d\u5b66\u6982\u7387</p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u6700\u540e\u4e00\u5c42</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u751f\u6210\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Get attention reconstruction loss </p>\n": "<p>\u5f15\u8d77\u6ce8\u610f\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Get memories </p>\n": "<p>\u83b7\u5f97\u56de\u5fc6</p>\n",
|
||||
"<p>Get memory and compressed memory </p>\n": "<p>\u83b7\u53d6\u5185\u5b58\u548c\u538b\u7f29\u5185\u5b58</p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u9884\u6d4b\uff08\u8d2a\u5a6a\uff09</p>\n",
|
||||
"<p>If the configurations specify not to use memory </p>\n": "<p>\u5982\u679c\u914d\u7f6e\u6307\u5b9a\u4e0d\u4f7f\u7528\u5185\u5b58</p>\n",
|
||||
"<p>If there are no old compressed memories </p>\n": "<p>\u5982\u679c\u6ca1\u6709\u65e7\u7684\u538b\u7f29\u8bb0\u5fc6</p>\n",
|
||||
"<p>Iterate through memories of each layer. </p>\n": "<p>\u904d\u5386\u6bcf\u5c42\u7684\u8bb0\u5fc6\u3002</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u8bb0\u5f55\u6bcf\u4e2a\u7eaa\u5143\u6700\u540e\u4e00\u6279\u7684\u6a21\u578b\u53c2\u6570\u548c\u68af\u5ea6</p>\n",
|
||||
"<p>Masks </p>\n": "<p>\u53e3\u7f69</p>\n",
|
||||
"<p>Merge and compress memory </p>\n": "<p>\u5408\u5e76\u548c\u538b\u7f29\u5185\u5b58</p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u5c06\u6570\u636e\u79fb\u52a8\u5230\u8bbe\u5907</p>\n",
|
||||
"<p>Move to device </p>\n": "<p>\u79fb\u81f3\u8bbe\u5907</p>\n",
|
||||
"<p>No memories are compressed if the number of memories is less than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u5185\u5b58\u6570\u91cf\u5c11\u4e8e<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u6ce8\u610f\u5934\u6570\u91cf</p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN \u9690\u85cf\u5c42\u4e2d\u7684\u8981\u7d20\u6570\u91cf</p>\n",
|
||||
"<p>Number of memories to compress <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8981\u538b\u7f29\u7684\u5185\u5b58\u6570\u91cf<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of memories to keep </p>\n": "<p>\u8981\u4fdd\u7559\u7684\u8bb0\u5fc6\u6570\u91cf</p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u53d8\u538b\u5668\u5c42\u6570</p>\n",
|
||||
"<p>Only feed the last character to model in next iteration, rest will go in as memories </p>\n": "<p>\u5728\u4e0b\u4e00\u6b21\u8fed\u4ee3\u4e2d\u53ea\u5582\u6700\u540e\u4e00\u4e2a\u89d2\u8272\u8fdb\u884c\u5efa\u6a21\uff0c\u5176\u4f59\u90e8\u5206\u5c06\u4f5c\u4e3a\u8bb0\u5fc6\u8fdb\u53bb</p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u6253\u5370\u91c7\u6837\u8f93\u51fa</p>\n",
|
||||
"<p>Return memories and the memories that were compressed. Memories that were compressed are needed for the reconstruction loss computation. </p>\n": "<p>\u8fd4\u56de\u88ab\u538b\u7f29\u7684\u8bb0\u5fc6\u548c\u8bb0\u5fc6\u3002\u91cd\u5efa\u635f\u5931\u8ba1\u7b97\u9700\u8981\u88ab\u538b\u7f29\u7684\u8bb0\u5fc6\u3002</p>\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u7528\u5b83\u7a7f\u8fc7\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Run the model </p>\n": "<p>\u8fd0\u884c\u6a21\u578b</p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>\u6837\u672c 25 \u4e2a\u4ee3\u5e01</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u4fdd\u5b58\u8ddf\u8e2a\u7684\u6307\u6807</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u8bbe\u7f6e\u8ddf\u8e2a\u5668\u914d\u7f6e</p>\n",
|
||||
"<p>Split the memories at <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5728\u4ee5\u4e0b\u4f4d\u7f6e\u62c6\u5206\u8bb0\u5fc6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u542f\u52a8\u63d0\u793a</p>\n",
|
||||
"<p>State module to maintain memories when switching between training and validation </p>\n": "<p>\u72b6\u6001\u6a21\u5757\u7528\u4e8e\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u65f6\u4fdd\u6301\u8bb0\u5fc6</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u91c7\u53d6\u4f18\u5316\u5668\u6b65\u9aa4</p>\n",
|
||||
"<p>This will keep the accuracy metric stats and memories separate for training and validation. </p>\n": "<p>\u8fd9\u5c06\u4f7f\u7cbe\u5ea6\u6307\u6807\u7edf\u8ba1\u6570\u636e\u548c\u8bb0\u5fc6\u5206\u5f00\uff0c\u4ee5\u4fbf\u8bad\u7ec3\u548c\u9a8c\u8bc1\u3002</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u6a21\u5757</p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u5c06\u63d0\u793a\u7b26\u53f7\u5316</p>\n",
|
||||
"<p>Total length of the memory and compressed memory (for masks) </p>\n": "<p>\u5185\u5b58\u548c\u538b\u7f29\u5185\u5b58\u7684\u603b\u957f\u5ea6\uff08\u7528\u4e8e\u63a9\u7801\uff09</p>\n",
|
||||
"<p>Track attention reconstruction loss </p>\n": "<p>\u8ffd\u8e2a\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u8bad\u7ec3\u6a21\u578b</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Truncate old memories </p>\n": "<p>\u622a\u65ad\u65e7\u7684\u8bb0\u5fc6</p>\n",
|
||||
"<p>Update and compress memory </p>\n": "<p>\u66f4\u65b0\u548c\u538b\u7f29\u5185\u5b58</p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u5728\u8bad\u7ec3\u6a21\u5f0f\u4e0b\u66f4\u65b0\u5168\u5c40\u6b65\u957f\uff08\u5904\u7406\u7684\u4ee4\u724c\u6570\uff09</p>\n",
|
||||
"<p>Update memories </p>\n": "<p>\u66f4\u65b0\u8bb0\u5fc6</p>\n",
|
||||
"<p>Update the memories </p>\n": "<p>\u66f4\u65b0\u8bb0\u5fc6</p>\n",
|
||||
"<p>Use only the subsequent mask otherwise </p>\n": "<p>\u5426\u5219\uff0c\u4ec5\u4f7f\u7528\u540e\u7eed\u7684\u63a9\u7801</p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u662f\u5426\u6355\u83b7\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>memory </p>\n": "<p>\u8bb0\u5fc6</p>\n",
|
||||
"Compressive Transformer Experiment": "\u538b\u7f29\u53d8\u538b\u5668\u5b9e\u9a8c",
|
||||
"This experiment trains a compressive transformer model on tiny Shakespeare dataset.": "\u8fd9\u4e2a\u5b9e\u9a8c\u5728\u5fae\u5c0f\u7684\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u4e00\u4e2a\u538b\u7f29\u53d8\u538b\u5668\u6a21\u578b\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"<h1>Configurable Transformer Components</h1>\n": "<h1>\u8a2d\u5b9a\u53ef\u80fd\u306a\u5909\u5727\u5668\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8</h1>\n",
|
||||
"<h2>GLU Variants</h2>\n<p>These are variants with gated hidden layers for the FFN as introduced in paper <a href=\"https://arxiv.org/abs/2002.05202\">GLU Variants Improve Transformer</a>. We have omitted the bias terms as specified in the paper. </p>\n": "<h2>GLU \u30d0\u30ea\u30a2\u30f3\u30c8</h2>\n<p>\u3053\u308c\u3089\u306f\u3001<a href=\"https://arxiv.org/abs/2002.05202\">\u7d19\u306eGLU\u30d0\u30ea\u30a2\u30f3\u30c8\u6539\u826f\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3067\u7d39\u4ecb\u3055\u308c\u3066\u3044\u308b\u3088\u3046\u306b\u3001FFN\u7528\u306e\u30b2\u30fc\u30c8\u96a0\u308c\u5c64\u3092\u5099\u3048\u305f\u30d0\u30ea\u30a2\u30f3\u30c8\u3067\u3059</a>\u3002\u8ad6\u6587\u3067\u660e\u8a18\u3055\u308c\u3066\u3044\u308b\u30d0\u30a4\u30a2\u30b9\u7528\u8a9e\u306f\u7701\u7565\u3057\u3066\u3044\u307e\u3059</p>\u3002\n",
|
||||
"<h3>FFN with Bilinear hidden layer</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>\u30d0\u30a4\u30ea\u30cb\u30a2\u96a0\u308c\u5c64\u4ed8\u304dFFN</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>FFN with GELU gate</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>GELU \u30b2\u30fc\u30c8\u4ed8\u304dFFN</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>FFN with Gated Linear Units</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>\u30b2\u30fc\u30c8\u4ed8\u304d\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8\u4ed8\u304dFFN</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>FFN with ReLU gate</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>RelU \u30b2\u30fc\u30c8\u4ed8\u304d FN</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>FFN with Swish gate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> </p>\n": "<h3>FFN\uff08\u30b9\u30a6\u30a3\u30c3\u30b7\u30e5\u30b2\u30fc\u30c8\u4ed8\u304d\uff09</h3>\n<p><span translate=no>_^_0_^_</span>\u3069\u3053 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<h3>Fixed Positional Embeddings</h3>\n<p>Source embedding with fixed positional encodings</p>\n": "<h3>\u56fa\u5b9a\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f</h3>\n<p>\u56fa\u5b9a\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30bd\u30fc\u30b9\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<h3>GELU activation</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span></p>\n<p>It was introduced in paper <a href=\"https://arxiv.org/abs/1606.08415\">Gaussian Error Linear Units</a>.</p>\n": "<h3>GELU \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3</h3>\n<p><span translate=no>_^_0_^_</span>\u3069\u3053 <span translate=no>_^_1_^_</span></p>\n<p><a href=\"https://arxiv.org/abs/1606.08415\">\u30ac\u30a6\u30b9\u8aa4\u5dee\u7dda\u5f62\u5358\u4f4d\u306e\u8ad6\u6587\u3067\u7d39\u4ecb\u3055\u308c\u307e\u3057\u305f</a>\u3002</p>\n",
|
||||
"<h3>Learned Positional Embeddings</h3>\n<p>Source embedding with learned positional encodings</p>\n": "<h3>\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u3092\u5b66\u3093\u3060</h3>\n<p>\u5b66\u7fd2\u3057\u305f\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30bd\u30fc\u30b9\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<h3>Multi-head Attention</h3>\n": "<h3>\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3</h3>\n",
|
||||
"<h3>No Positional Embeddings</h3>\n<p>Source embedding without positional encodings</p>\n": "<h3>\u4f4d\u7f6e\u6307\u5b9a\u57cb\u3081\u8fbc\u307f\u306a\u3057</h3>\n<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306a\u3057\u306e\u30bd\u30fc\u30b9\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<h3>ReLU activation</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>ReLU \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>Relative Multi-head Attention</h3>\n": "<h3>\u76f8\u5bfe\u7684\u306a\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3</h3>\n",
|
||||
"<p> <a id=\"FFN\"></a></p>\n<h2>FFN Configurations</h2>\n<p>Creates a Position-wise FeedForward Network defined in <a href=\"feed_forward.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<p><a id=\"FFN\"></a></p>\n<h2>FFN \u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u4f4d\u7f6e\u5358\u4f4d\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u4f5c\u6210\u3057\u307e\u3059\u3002<a href=\"feed_forward.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<p> <a id=\"TransformerConfigs\"></a></p>\n<h2>Transformer Configurations</h2>\n<p>This defines configurations for a transformer. The configurations are calculate using option functions. These are lazy loaded and therefore only the necessary modules are calculated.</p>\n": "<p><a id=\"TransformerConfigs\"></a></p>\n<h2>\u5909\u5727\u5668\u69cb\u6210</h2>\n<p>\u3053\u308c\u306f\u5909\u5727\u5668\u306e\u69cb\u6210\u3092\u5b9a\u7fa9\u3057\u307e\u3059\u3002\u69cb\u6210\u306f\u30aa\u30d7\u30b7\u30e7\u30f3\u95a2\u6570\u3092\u4f7f\u7528\u3057\u3066\u8a08\u7b97\u3055\u308c\u307e\u3059\u3002\u3053\u308c\u3089\u306f\u9045\u5ef6\u30ed\u30fc\u30c9\u3055\u308c\u308b\u305f\u3081\u3001\u5fc5\u8981\u306a\u30e2\u30b8\u30e5\u30fc\u30eb\u3060\u3051\u304c\u8a08\u7b97\u3055\u308c\u307e\u3059</p>\u3002\n",
|
||||
"<p> Create feedforward layer configurations</p>\n": "<p>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64\u69cb\u6210\u306e\u4f5c\u6210</p>\n",
|
||||
"<p> Decoder layer</p>\n": "<p>\u30c7\u30b3\u30fc\u30c0\u30fc\u5c64</p>\n",
|
||||
"<p> Decoder</p>\n": "<p>\u30c7\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p> Encoder layer</p>\n": "<p>\u30a8\u30f3\u30b3\u30fc\u30c0\u5c64</p>\n",
|
||||
"<p> Encoder</p>\n": "<p>\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p> Initialize a <a href=\"feed_forward.html\">feed forward network</a></p>\n": "<p><a href=\"feed_forward.html\">\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u521d\u671f\u5316</a></p>\n",
|
||||
"<p> Logit generator</p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u30fb\u30b8\u30a7\u30cd\u30ec\u30fc\u30bf\u30fc</p>\n",
|
||||
"<p> Target embedding with fixed positional encodings</p>\n": "<p>\u56fa\u5b9a\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30bf\u30fc\u30b2\u30c3\u30c8\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<p> Target embedding with learned positional encodings</p>\n": "<p>\u5b66\u7fd2\u3057\u305f\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30bf\u30fc\u30b2\u30c3\u30c8\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<p>Activation in position-wise feedforward layer </p>\n": "<p>\u4f4d\u7f6e\u5358\u4f4d\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64\u3067\u306e\u6d3b\u6027\u5316</p>\n",
|
||||
"<p>Configurable Feedforward Layer </p>\n": "<p>\u8a2d\u5b9a\u53ef\u80fd\u306a\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64</p>\n",
|
||||
"<p>Decoder layer </p>\n": "<p>\u30c7\u30b3\u30fc\u30c0\u30fc\u5c64</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8131\u843d\u78ba\u7387</p>\n",
|
||||
"<p>Embedding layer for source </p>\n": "<p>\u30bd\u30fc\u30b9\u306e\u57cb\u3081\u8fbc\u307f\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Embedding layer for target (for decoder) </p>\n": "<p>\u30bf\u30fc\u30b2\u30c3\u30c8\u7528\u57cb\u3081\u8fbc\u307f\u30ec\u30a4\u30e4\u30fc (\u30c7\u30b3\u30fc\u30c0\u30fc\u7528)</p>\n",
|
||||
"<p>Encoder consisting of multiple decoder layers </p>\n": "<p>\u8907\u6570\u306e\u30c7\u30b3\u30fc\u30c0\u30fc\u5c64\u3067\u69cb\u6210\u3055\u308c\u308b\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Encoder consisting of multiple encoder layers </p>\n": "<p>\u8907\u6570\u306e\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u5c64\u3067\u69cb\u6210\u3055\u308c\u308b\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Encoder layer </p>\n": "<p>\u30a8\u30f3\u30b3\u30fc\u30c0\u5c64</p>\n",
|
||||
"<p>Encoder-decoder </p>\n": "<p>\u30a8\u30f3\u30b3\u30fc\u30c0/\u30c7\u30b3\u30fc\u30c0</p>\n",
|
||||
"<p>Logit generator for prediction </p>\n": "<p>\u4e88\u6e2c\u7528\u30ed\u30b8\u30c3\u30c8\u30fb\u30b8\u30a7\u30cd\u30ec\u30fc\u30bf\u30fc</p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u6570</p>\n",
|
||||
"<p>Number of features in in the hidden layer </p>\n": "<p>\u96a0\u308c\u30ec\u30a4\u30e4\u30fc\u306b\u542b\u307e\u308c\u308b\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u6570</p>\n",
|
||||
"<p>Number of features in the embedding </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u306b\u542b\u307e\u308c\u308b\u6a5f\u80fd\u306e\u6570</p>\n",
|
||||
"<p>Number of layers </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u6570</p>\n",
|
||||
"<p>Number of tokens in the source vocabulary (for token embeddings) </p>\n": "<p>\u30bd\u30fc\u30b9\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u306e\u30c8\u30fc\u30af\u30f3\u6570 (\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u7528)</p>\n",
|
||||
"<p>Number of tokens in the target vocabulary (to generate logits for prediction) </p>\n": "<p>\u30bf\u30fc\u30b2\u30c3\u30c8\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u5185\u306e\u30c8\u30fc\u30af\u30f3\u306e\u6570 (\u4e88\u6e2c\u7528\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u305f\u3081)</p>\n",
|
||||
"<p>Position-wise feedforward layer </p>\n": "<p>\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64</p>\n",
|
||||
"<p>Predefined GLU variants </p>\n": "<p>\u5b9a\u7fa9\u6e08\u307f\u306e GLU \u30d0\u30ea\u30a2\u30f3\u30c8</p>\n",
|
||||
"<p>The decoder memory attention </p>\n": "<p>\u30c7\u30b3\u30fc\u30c0\u30e1\u30e2\u30ea\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>The decoder self attention </p>\n": "<p>\u30c7\u30b3\u30fc\u30c0\u30fc\u306e\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>The encoder self attention </p>\n": "<p>\u30a8\u30f3\u30b3\u30fc\u30c0\u306e\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Transformer embedding size </p>\n": "<p>\u5909\u5727\u5668\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Whether the FFN layer should be gated </p>\n": "<p>FFN \u30ec\u30a4\u30e4\u30fc\u3092\u30b2\u30fc\u30c8\u3059\u3079\u304d\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>Whether the first fully connected layer should have a learnable bias </p>\n": "<p>\u6700\u521d\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u4ed8\u3051\u308b\u3079\u304d\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>Whether the fully connected layer for the gate should have a learnable bias </p>\n": "<p>\u30b2\u30fc\u30c8\u306e\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u8a2d\u3051\u308b\u3079\u304d\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>Whether the second fully connected layer should have a learnable bias </p>\n": "<p>2 \u756a\u76ee\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u8a2d\u5b9a\u3059\u3079\u304d\u304b\u3069\u3046\u304b</p>\n",
|
||||
"Configurable Transformer Components": "\u8a2d\u5b9a\u53ef\u80fd\u306a\u5909\u5727\u5668\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8",
|
||||
"These are configurable components that can be re-used quite easily.": "\u3053\u308c\u3089\u306f\u8a2d\u5b9a\u53ef\u80fd\u306a\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8\u3067\u3001\u7c21\u5358\u306b\u518d\u5229\u7528\u3067\u304d\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"<h1>Configurable Transformer Components</h1>\n": "<h1>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad\u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0d82\u0dbb\u0da0\u0d9a</h1>\n",
|
||||
"<h2>GLU Variants</h2>\n<p>These are variants with gated hidden layers for the FFN as introduced in paper <a href=\"https://arxiv.org/abs/2002.05202\">GLU Variants Improve Transformer</a>. We have omitted the bias terms as specified in the paper. </p>\n": "<h2>GLU\u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf</h2>\n<p>\u0db8\u0dda\u0dc0\u0dcf\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd3 \u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0d91\u0dc6\u0dca\u0d91\u0dc6\u0dca\u0d91\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4 \u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0dc3\u0dc4\u0dd2\u0dad \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf \u0dc0\u0dda <a href=\"https://arxiv.org/abs/2002.05202\">GLU \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a>. \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dbd \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0d85\u0db4\u0dd2 \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 \u0db1\u0dd2\u0dba\u0db8\u0dba\u0db1\u0dca \u0db8\u0d9f \u0dc4\u0dd0\u0dbb\u0dd2 \u0d87\u0dad. </p>\n",
|
||||
"<h3>FFN with Bilinear hidden layer</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>\u0db6\u0dd2\u0dbd\u0dd3\u0db1\u0dd2\u0dba\u0dbb\u0dca\u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dad\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0d9a\u0dca \u0dc3\u0dc4\u0dd2\u0dad FFN</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<h3>FFN with GELU gate</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>GELU\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dc0 \u0dc3\u0dc4\u0dd2\u0dad FFN</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<h3>FFN with Gated Linear Units</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0db8\u0d9f FFN</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<h3>FFN with ReLU gate</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>RelU\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dc0 \u0dc3\u0db8\u0d9f FFN</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<h3>FFN with Swish gate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> </p>\n": "<h3>\u0dc3\u0dca\u0dc0\u0dd2\u0dc2\u0dca\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dc0 \u0dc3\u0db8\u0d9f \u0d91\u0dc6\u0dca\u0d91\u0dc6\u0dca\u0d91\u0db1\u0dca</h3>\n<p><span translate=no>_^_0_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<h3>Fixed Positional Embeddings</h3>\n<p>Source embedding with fixed positional encodings</p>\n": "<h3>\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca</h3>\n<p>\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab \u0dc3\u0db8\u0d9f \u0db4\u0dca\u0dbb\u0db7\u0dc0 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8</p>\n",
|
||||
"<h3>GELU activation</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span></p>\n<p>It was introduced in paper <a href=\"https://arxiv.org/abs/1606.08415\">Gaussian Error Linear Units</a>.</p>\n": "<h3>GELU\u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</h3>\n<p><span translate=no>_^_0_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf <span translate=no>_^_1_^_</span></p>\n<p>\u0d91\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://arxiv.org/abs/1606.08415\">Gaussian \u0daf\u0ddd\u0dc2 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a</a>\u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0dbd\u0daf\u0dd3. </p>\n",
|
||||
"<h3>Learned Positional Embeddings</h3>\n<p>Source embedding with learned positional encodings</p>\n": "<h3>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0d89\u0d9c\u0dd9\u0db1</h3>\n<p>\u0d8b\u0d9c\u0dad\u0dca\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab \u0dc3\u0db8\u0d9f \u0db4\u0dca\u0dbb\u0db7\u0dc0\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8</p>\n",
|
||||
"<h3>Multi-head Attention</h3>\n": "<h3>\u0db6\u0dc4\u0dd4-\u0dc4\u0dd2\u0dc3\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba</h3>\n",
|
||||
"<h3>No Positional Embeddings</h3>\n<p>Source embedding without positional encodings</p>\n": "<h3>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad</h3>\n<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dad\u0ddc\u0dbb\u0dc0 \u0db4\u0dca\u0dbb\u0db7\u0dc0 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8</p>\n",
|
||||
"<h3>ReLU activation</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>Relu\u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>Relative Multi-head Attention</h3>\n": "<h3>\u0dc3\u0dcf\u0db4\u0dda\u0d9a\u0dca\u0dc2\u0db6\u0dc4\u0dd4-\u0dc4\u0dd2\u0dc3 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba</h3>\n",
|
||||
"<p> <a id=\"FFN\"></a></p>\n<h2>FFN Configurations</h2>\n<p>Creates a Position-wise FeedForward Network defined in <a href=\"feed_forward.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<p> <a id=\"FFN\"></a></p>\n<h2>FFN\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h2>\n<p>\u0d85\u0dbb\u0dca\u0dae\u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca Feed\u0d89\u0daf\u0dd2\u0dbb\u0dd2 \u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0dca \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb\u0dba\u0dd2 <a href=\"feed_forward.html\"><span translate=no>_^_0_^_</span></a>. </p>\n",
|
||||
"<p> <a id=\"TransformerConfigs\"></a></p>\n<h2>Transformer Configurations</h2>\n<p>This defines configurations for a transformer. The configurations are calculate using option functions. These are lazy loaded and therefore only the necessary modules are calculated.</p>\n": "<p> <a id=\"TransformerConfigs\"></a></p>\n<h2>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dba\u0dd2. \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0dc0\u0dd2\u0d9a\u0dbd\u0dca\u0db4 \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0db1\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd2\u0db1\u0dca \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0dd0\u0db6\u0dda. \u0db8\u0dda\u0dc0\u0dcf \u0d9a\u0db8\u0dca\u0db8\u0dd0\u0dbd\u0dd2 \u0dbd\u0dd9\u0dc3 \u0db4\u0da7\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd \u0db4\u0db8\u0dab\u0d9a\u0dca \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0dd0\u0db6\u0dda. </p>\n",
|
||||
"<p> Create feedforward layer configurations</p>\n": "<p> \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab\u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p> Decoder layer</p>\n": "<p> \u0dc3\u0dca\u0dad\u0dbb\u0dba\u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba</p>\n",
|
||||
"<p> Decoder</p>\n": "<p> \u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba</p>\n",
|
||||
"<p> Encoder layer</p>\n": "<p> \u0d86\u0d9a\u0dda\u0dad\u0d9a\u0dba\u0dc3\u0dca\u0dae\u0dbb\u0dba</p>\n",
|
||||
"<p> Encoder</p>\n": "<p> \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</p>\n",
|
||||
"<p> Initialize a <a href=\"feed_forward.html\">feed forward network</a></p>\n": "<p> <a href=\"feed_forward.html\">\u0d86\u0dc4\u0dcf\u0dbb \u0d89\u0daf\u0dd2\u0dbb\u0dd2 \u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0dca</a>\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p> Logit generator</p>\n": "<p> \u0dbd\u0ddc\u0d9c\u0dd2\u0db1\u0dca\u0dc0\u0db1\u0dca\u0db1 \u0da2\u0db1\u0d9a\u0dba</p>\n",
|
||||
"<p> Target embedding with fixed positional encodings</p>\n": "<p> \u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0db1 \u0d9a\u0dca\u0dbb\u0db8 \u0dc3\u0db8\u0d9f \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0d89\u0dbd\u0d9a\u0dca\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p> Target embedding with learned positional encodings</p>\n": "<p> \u0d8b\u0d9c\u0dad\u0dca\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0db1 \u0d9a\u0dca\u0dbb\u0db8 \u0dc3\u0db8\u0d9f \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0d89\u0dbd\u0d9a\u0dca\u0d9a \u0d9a\u0dbb \u0d9c\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p>Activation in position-wise feedforward layer </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca\u0db4\u0ddd\u0dc2\u0d9a \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Configurable Feedforward Layer </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad\u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Decoder layer </p>\n": "<p>\u0dc3\u0dca\u0dad\u0dbb\u0dba\u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba </p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u0d85\u0dad\u0dc4\u0dd0\u0dbb\u0daf\u0dd0\u0db8\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Embedding layer for source </p>\n": "<p>\u0db4\u0dca\u0dbb\u0db7\u0dc0\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 </p>\n",
|
||||
"<p>Embedding layer for target (for decoder) </p>\n": "<p>\u0d89\u0dbd\u0d9a\u0dca\u0d9a\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 (\u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba \u0dc3\u0db3\u0dc4\u0dcf) </p>\n",
|
||||
"<p>Encoder consisting of multiple decoder layers </p>\n": "<p>\u0db6\u0dc4\u0dd4\u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a \u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0db1\u0dca\u0dc0\u0dd2\u0dad \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Encoder consisting of multiple encoder layers </p>\n": "<p>\u0db6\u0dc4\u0dd4\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca \u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0db1\u0dca\u0dc0\u0dd2\u0dad \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Encoder layer </p>\n": "<p>\u0d86\u0d9a\u0dda\u0dad\u0d9a\u0dba\u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Encoder-decoder </p>\n": "<p>\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba-\u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba </p>\n",
|
||||
"<p>Logit generator for prediction </p>\n": "<p>\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0dbd\u0ddc\u0d9c\u0dd2\u0db1\u0dca \u0dc0\u0db1\u0dca\u0db1 \u0da2\u0db1\u0d9a\u0dba </p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of features in in the hidden layer </p>\n": "<p>\u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4\u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0d87\u0dad\u0dd2 \u0dbd\u0d9a\u0dca\u0dc2\u0dab \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of features in the embedding </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda\u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of layers </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dbb\u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of tokens in the source vocabulary (for token embeddings) </p>\n": "<p>\u0db4\u0dca\u0dbb\u0db7\u0dc0\u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0\u0dda \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1 (\u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0db3\u0dc4\u0dcf) </p>\n",
|
||||
"<p>Number of tokens in the target vocabulary (to generate logits for prediction) </p>\n": "<p>\u0d89\u0dbd\u0d9a\u0dca\u0d9a\u0d9c\u0dad\u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0\u0dda \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1 (\u0db4\u0dd4\u0dbb\u0ddd\u0d9a\u0dae\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7) </p>\n",
|
||||
"<p>Position-wise feedforward layer </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca\u0db4\u0ddd\u0dc2\u0d9a \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Predefined GLU variants </p>\n": "<p>\u0db4\u0dd6\u0dbb\u0dca\u0dc0\u0db1\u0dd2\u0dc1\u0dca\u0da0\u0dd2\u0dad GLU \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf </p>\n",
|
||||
"<p>The decoder memory attention </p>\n": "<p>\u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0db8\u0dad\u0d9a \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba </p>\n",
|
||||
"<p>The decoder self attention </p>\n": "<p>\u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba </p>\n",
|
||||
"<p>The encoder self attention </p>\n": "<p>\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba </p>\n",
|
||||
"<p>Transformer embedding size </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Whether the FFN layer should be gated </p>\n": "<p>FFN\u0dc3\u0dca\u0dad\u0dbb\u0dba \u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4 \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether the first fully connected layer should have a learnable bias </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0da7 \u0d89\u0d9c\u0dd9\u0db1 \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2 \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0\u0d9a\u0dca \u0dad\u0dd2\u0db6\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether the fully connected layer for the gate should have a learnable bias </p>\n": "<p>\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0d89\u0d9c\u0dd9\u0db1 \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2 \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0\u0d9a\u0dca \u0dad\u0dd2\u0db6\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether the second fully connected layer should have a learnable bias </p>\n": "<p>\u0daf\u0dd9\u0dc0\u0db1\u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0d89\u0d9c\u0dd9\u0db1 \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2 \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0\u0d9a\u0dca \u0dad\u0dd2\u0db6\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"Configurable Transformer Components": "\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0d82\u0dbb\u0da0\u0d9a",
|
||||
"These are configurable components that can be re-used quite easily.": "\u0db8\u0dda\u0dc0\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0dc3\u0d82\u0dbb\u0da0\u0d9a \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0d92\u0dc0\u0dcf \u0db4\u0dc4\u0dc3\u0dd4\u0dc0\u0dd9\u0db1\u0dca \u0db1\u0dd0\u0dc0\u0dad \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba."
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"<h1>Configurable Transformer Components</h1>\n": "<h1>\u53ef\u914d\u7f6e\u7684 Transformer \u7ec4\u4ef6</h1>\n",
|
||||
"<h2>GLU Variants</h2>\n<p>These are variants with gated hidden layers for the FFN as introduced in paper <a href=\"https://arxiv.org/abs/2002.05202\">GLU Variants Improve Transformer</a>. We have omitted the bias terms as specified in the paper. </p>\n": "<h2>GLU \u53d8\u4f53</h2>\n<p>\u8fd9\u4e9b\u662f\u5728\u8bba\u6587 <a href=\"https://arxiv.org/abs/2002.05202\">\u300a GLU Variants Improve Transformer \u300b</a>\u4e2d\u5305\u542b\u7684\u5404\u79cd\u5e26\u95e8\u63a7\u9690\u85cf\u5c42\u7684 FFN \u53d8\u4f53\u3002\u6211\u4eec\u5df2\u6309\u7167\u8bba\u6587\u89c4\u5b9a\u7701\u7565\u4e86\u504f\u7f6e\u9879\u3002</p>\n",
|
||||
"<h3>FFN with Bilinear hidden layer</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>\u5e26\u53cc\u7ebf\u6027\u9690\u85cf\u5c42\u7684 FFN</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>FFN with GELU gate</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>\u5e26 GELU \u95e8\u7684 FFN</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>FFN with Gated Linear Units</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>\u5e26\u95e8\u63a7\u7ebf\u6027\u5355\u5143\u7684 FFN</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>FFN with ReLU gate</h3>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<h3>\u5e26 ReLU \u95e8\u7684 FFN</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>FFN with Swish gate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> </p>\n": "<h3>\u5e26 Swish \u95e8\u7684 FFN</h3>\n<p><span translate=no>_^_0_^_</span>\u5176\u4e2d\uff0c<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<h3>Fixed Positional Embeddings</h3>\n<p>Source embedding with fixed positional encodings</p>\n": "<h3>\u56fa\u5b9a\u4f4d\u7f6e\u5d4c\u5165</h3>\n<p>\u4f7f\u7528\u56fa\u5b9a\u4f4d\u7f6e\u7f16\u7801\u8fdb\u884c\u6e90\u5d4c\u5165</p>\n",
|
||||
"<h3>GELU activation</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span></p>\n<p>It was introduced in paper <a href=\"https://arxiv.org/abs/1606.08415\">Gaussian Error Linear Units</a>.</p>\n": "<h3>GELU \u6fc0\u6d3b\u51fd\u6570</h3>\n<p><span translate=no>_^_0_^_</span>\u5176\u4e2d\uff0c<span translate=no>_^_1_^_</span></p>\n<p>\u8fd9\u662f\u5728\u8bba\u6587<a href=\"https://arxiv.org/abs/1606.08415\">\u300a Gaussian Error Linear Units \u300b</a>\u4e2d\u4ecb\u7ecd\u7684\u3002</p>\n",
|
||||
"<h3>Learned Positional Embeddings</h3>\n<p>Source embedding with learned positional encodings</p>\n": "<h3>\u53ef\u5b66\u4e60\u7684\u4f4d\u7f6e\u5d4c\u5165</h3>\n<p>\u4f7f\u7528\u53ef\u5b66\u4e60\u7684\u4f4d\u7f6e\u7f16\u7801\u8fdb\u884c\u5d4c\u5165</p>\n",
|
||||
"<h3>Multi-head Attention</h3>\n": "<h3>\u591a\u5934\u6ce8\u610f\u529b</h3>\n",
|
||||
"<h3>No Positional Embeddings</h3>\n<p>Source embedding without positional encodings</p>\n": "<h3>\u65e0\u4f4d\u7f6e\u5d4c\u5165</h3>\n<p>\u6ca1\u6709\u4f4d\u7f6e\u7f16\u7801\u7684\u6e90\u5d4c\u5165</p>\n",
|
||||
"<h3>ReLU activation</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>ReLU \u6fc0\u6d3b\u51fd\u6570</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<h3>Relative Multi-head Attention</h3>\n": "<h3>\u76f8\u5bf9\u591a\u5934\u6ce8\u610f\u529b</h3>\n",
|
||||
"<p> <a id=\"FFN\"></a></p>\n<h2>FFN Configurations</h2>\n<p>Creates a Position-wise FeedForward Network defined in <a href=\"feed_forward.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<p><a id=\"FFN\"></a></p>\n<h2>FFN \u914d\u7f6e</h2>\n<p>\u5728<a href=\"feed_forward.html\"><span translate=no>_^_0_^_</span></a>\u4e2d\u5b9a\u4e49\u4e86\u4e00\u4e2a\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc\u3002</p>\n",
|
||||
"<p> <a id=\"TransformerConfigs\"></a></p>\n<h2>Transformer Configurations</h2>\n<p>This defines configurations for a transformer. The configurations are calculate using option functions. These are lazy loaded and therefore only the necessary modules are calculated.</p>\n": "<p><a id=\"TransformerConfigs\"></a></p>\n<h2>Transformer \u914d\u7f6e</h2>\n<p>\u8fd9\u5b9a\u4e49\u4e86 Transformer \u7684\u914d\u7f6e\u3002\u8fd9\u4e9b\u914d\u7f6e\u662f\u901a\u8fc7\u53ef\u9009\u62e9\u7684\u51fd\u6570\u8fdb\u884c\u8ba1\u7b97\u7684\u3002\u5b83\u4eec\u662f\u60f0\u6027\u52a0\u8f7d\u7684\uff0c\u56e0\u6b64\u53ea\u6709\u5fc5\u8981\u7684\u6a21\u5757\u624d\u4f1a\u88ab\u8ba1\u7b97\u3002</p>\n",
|
||||
"<p> Create feedforward layer configurations</p>\n": "<p>\u521b\u5efa\u524d\u9988\u5c42\u914d\u7f6e</p>\n",
|
||||
"<p> Decoder layer</p>\n": "<p>\u89e3\u7801\u5668\u5c42</p>\n",
|
||||
"<p> Decoder</p>\n": "<p>\u89e3\u7801\u5668</p>\n",
|
||||
"<p> Encoder layer</p>\n": "<p>\u7f16\u7801\u5668\u5c42</p>\n",
|
||||
"<p> Encoder</p>\n": "<p>\u7f16\u7801\u5668</p>\n",
|
||||
"<p> Initialize a <a href=\"feed_forward.html\">feed forward network</a></p>\n": "<p>\u521d\u59cb\u5316<a href=\"feed_forward.html\">\u524d\u9988\u7f51\u7edc</a></p>\n",
|
||||
"<p> Logit generator</p>\n": "<p>Logit \u751f\u6210\u5668</p>\n",
|
||||
"<p> Target embedding with fixed positional encodings</p>\n": "<p>\u4f7f\u7528\u56fa\u5b9a\u4f4d\u7f6e\u7f16\u7801\u8fdb\u884c\u76ee\u6807\u5d4c\u5165</p>\n",
|
||||
"<p> Target embedding with learned positional encodings</p>\n": "<p>\u4f7f\u7528\u53ef\u5b66\u4e60\u7684\u4f4d\u7f6e\u7f16\u7801\u8fdb\u884c\u76ee\u6807\u5d4c\u5165</p>\n",
|
||||
"<p>Activation in position-wise feedforward layer </p>\n": "<p>\u4f4d\u7f6e\u524d\u9988\u5c42\u4e2d\u7684\u6fc0\u6d3b\u51fd\u6570</p>\n",
|
||||
"<p>Configurable Feedforward Layer </p>\n": "<p>\u53ef\u914d\u7f6e\u7684\u524d\u9988\u5c42</p>\n",
|
||||
"<p>Decoder layer </p>\n": "<p>\u89e3\u7801\u5668\u5c42</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>Dropout \u7387</p>\n",
|
||||
"<p>Embedding layer for source </p>\n": "<p>\u6e90\u6570\u636e\u7684\u5d4c\u5165\u5c42</p>\n",
|
||||
"<p>Embedding layer for target (for decoder) </p>\n": "<p>\u76ee\u6807\u6570\u636e\u7684\u5d4c\u5165\u5c42\uff08\u7528\u4e8e\u89e3\u7801\u5668\uff09</p>\n",
|
||||
"<p>Encoder consisting of multiple decoder layers </p>\n": "<p>\u7531\u591a\u4e2a\u89e3\u7801\u5668\u5c42\u7ec4\u6210\u7684\u7f16\u7801\u5668</p>\n",
|
||||
"<p>Encoder consisting of multiple encoder layers </p>\n": "<p>\u7531\u591a\u4e2a\u7f16\u7801\u5668\u5c42\u7ec4\u6210\u7684\u7f16\u7801\u5668</p>\n",
|
||||
"<p>Encoder layer </p>\n": "<p>\u7f16\u7801\u5668\u5c42</p>\n",
|
||||
"<p>Encoder-decoder </p>\n": "<p>\u7f16\u7801\u5668-\u89e3\u7801\u5668</p>\n",
|
||||
"<p>Logit generator for prediction </p>\n": "<p>\u7528\u4e8e\u9884\u6d4b\u7684 Logit \u751f\u6210\u5668</p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u6ce8\u610f\u529b\u5934\u6570\u91cf</p>\n",
|
||||
"<p>Number of features in in the hidden layer </p>\n": "<p>\u9690\u85cf\u5c42\u4e2d\u7684\u7279\u5f81\u6570\u91cf</p>\n",
|
||||
"<p>Number of features in the embedding </p>\n": "<p>\u5d4c\u5165\u7684\u7279\u5f81\u6570\u91cf</p>\n",
|
||||
"<p>Number of layers </p>\n": "<p>\u5c42\u6570</p>\n",
|
||||
"<p>Number of tokens in the source vocabulary (for token embeddings) </p>\n": "<p>\u6e90\u8bcd\u6c47\u8868\u4e2d\u7684 token \u6570\u91cf\uff08\u7528\u4e8e token \u5d4c\u5165\uff09</p>\n",
|
||||
"<p>Number of tokens in the target vocabulary (to generate logits for prediction) </p>\n": "<p>\u76ee\u6807\u8bcd\u6c47\u8868\u4e2d\u7684 token \u6570\u91cf\uff08\u7528\u4e8e\u751f\u6210\u9884\u6d4b\u7684 logits \uff09</p>\n",
|
||||
"<p>Position-wise feedforward layer </p>\n": "<p>\u4f4d\u7f6e\u524d\u9988\u5c42</p>\n",
|
||||
"<p>Predefined GLU variants </p>\n": "<p>\u9884\u5b9a\u4e49\u7684 GLU \u53d8\u4f53</p>\n",
|
||||
"<p>The decoder memory attention </p>\n": "<p>\u89e3\u7801\u5668\u8bb0\u5fc6\u4e0e\u6ce8\u610f\u529b</p>\n",
|
||||
"<p>The decoder self attention </p>\n": "<p>\u89e3\u7801\u5668\u81ea\u6ce8\u610f\u529b</p>\n",
|
||||
"<p>The encoder self attention </p>\n": "<p>\u7f16\u7801\u5668\u81ea\u6ce8\u610f\u529b</p>\n",
|
||||
"<p>Transformer embedding size </p>\n": "<p>Transformer \u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Whether the FFN layer should be gated </p>\n": "<p>\u662f\u5426\u5e94\u5bf9 FFN \u5c42\u8fdb\u884c\u95e8\u63a7</p>\n",
|
||||
"<p>Whether the first fully connected layer should have a learnable bias </p>\n": "<p>\u7b2c\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e</p>\n",
|
||||
"<p>Whether the fully connected layer for the gate should have a learnable bias </p>\n": "<p>\u95e8\u63a7\u7684\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e</p>\n",
|
||||
"<p>Whether the second fully connected layer should have a learnable bias </p>\n": "<p>\u7b2c\u4e8c\u4e2a\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e</p>\n",
|
||||
"Configurable Transformer Components": "\u53ef\u914d\u7f6e Transformer \u7ec4\u4ef6",
|
||||
"These are configurable components that can be re-used quite easily.": "\u8fd9\u4e9b\u662f\u53ef\u914d\u7f6e\u7684\u7ec4\u4ef6\uff0c\u53ef\u4ee5\u5f88\u5bb9\u6613\u5730\u91cd\u590d\u4f7f\u7528\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"<h1>Train Fast Weights Transformer</h1>\n<p>This trains a fast weights transformer model for auto-regression.</p>\n<p>Here\u2019s a Colab notebook for training a fast weights transformer on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u9244\u9053\u7528\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u5909\u5727\u5668</h1>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001\u81ea\u52d5\u56de\u5e30\u7528\u306e\u9ad8\u901f\u91cd\u307f\u5909\u63db\u30e2\u30c7\u30eb\u304c\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3055\u308c\u307e\u3059\u3002</p>\n<p>\u3053\u308c\u306f\u3001Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306eColab\u30ce\u30fc\u30c8\u30d6\u30c3\u30af\u3067\u3059\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52d5\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u8a2d\u5b9a\u306f\u3001\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u305f\u3068\u304d\u306b\u4e0a\u66f8\u304d\u3067\u304d\u3001\u307e\u305f\u4e0a\u66f8\u304d\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p> Create <a href=\"index.html\">fast weights transformer</a>.</p>\n": "<p><a href=\"index.html\">\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u4f5c\u6210</a>\u3002</p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u3059\u308b\u8a2d\u5b9a\u306e\u8f9e\u66f8</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Embed the tokens </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u3092\u57cb\u3081\u8fbc\u3080</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Run it through the the transformer </p>\n": "<p>\u5909\u5727\u5668\u306b\u901a\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Run the training loop </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3092\u5b9f\u884c\u3059\u308b</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"This is training code with notes for a Fast Weights Transformer.": "\u3053\u308c\u306f\u3001\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306e\u30e1\u30e2\u4ed8\u304d\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3067\u3059\u3002",
|
||||
"Train Fast Weights Transformer": "\u9244\u9053\u7528\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u5909\u5727\u5668"
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"<h1>Train Fast Weights Transformer</h1>\n<p>This trains a fast weights transformer model for auto-regression.</p>\n<p>Here\u2019s a Colab notebook for training a fast weights transformer on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/928aadc0846c11eb85710242ac1c0002\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca\u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba \u0db6\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</h1>\n<p>\u0db8\u0dd9\u0dba\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dc3\u0dd4\u0d9c\u0dcf\u0db8\u0dd3 \u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2. </p>\n<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db8\u0dad \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0ddc\u0dbd\u0dd0\u0db6\u0dca \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0db4\u0ddc\u0dad\u0d9a\u0dca \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/928aadc0846c11eb85710242ac1c0002\"> <span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0d85\u0db4\u0dd2\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1 \u0dc0\u0dd2\u0da7 \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3 \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0d85\u0dad\u0dbb \u0d91\u0dba \u0d85\u0db0\u0dd2\u0d9a \u0dbd\u0dd9\u0dc3 \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0dc0\u0db1\u0dd4 \u0d87\u0dad</p>\n",
|
||||
"<p> Create <a href=\"index.html\">fast weights transformer</a>.</p>\n": "<p> <a href=\"index.html\">\u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca</a>\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf\u0dba\u0dcf\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc1\u0db6\u0dca\u0daf\u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dca </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Embed the tokens </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 </p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run it through the the transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0dc4\u0dbb\u0dc4\u0dcf \u0d91\u0dba \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run the training loop </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dbd\u0dd6\u0db4\u0dba \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"This is training code with notes for a Fast Weights Transformer.": "\u0db8\u0dd9\u0dba \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0dc3\u0dc4\u0dd2\u0dad \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dda\u0dad\u0dba\u0d9a\u0dd2.",
|
||||
"Train Fast Weights Transformer": "\u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba \u0db6\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca"
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"<h1>Train Fast Weights Transformer</h1>\n<p>This trains a fast weights transformer model for auto-regression.</p>\n<p>Here\u2019s a Colab notebook for training a fast weights transformer on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u706b\u8f66\u5feb\u901f\u91cd\u91cf\u53d8\u538b\u5668</h1>\n<p>\u8fd9\u4f1a\u8bad\u7ec3\u4e00\u4e2a\u7528\u4e8e\u81ea\u52a8\u56de\u5f52\u7684\u5feb\u901f\u6743\u91cd\u53d8\u6362\u5668\u6a21\u578b\u3002</p>\n<p>\u8fd9\u662f\u4e00\u672c Colab \u7b14\u8bb0\u672c\uff0c\u7528\u4e8e\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u5feb\u901f\u6743\u91cd\u8f6c\u6362\u5668\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52a8\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u5f53\u6211\u4eec\u5f00\u59cb\u5b9e\u9a8c\u65f6\uff0c\u9ed8\u8ba4\u914d\u7f6e\u53ef\u4ee5\u800c\u4e14\u5c06\u4f1a\u88ab\u8986\u76d6</p>\n",
|
||||
"<p> Create <a href=\"index.html\">fast weights transformer</a>.</p>\n": "<p>\u521b\u5efa<a href=\"index.html\">\u5feb\u901f\u6743\u91cd\u53d8\u538b\u5668</a>\u3002</p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u8981\u8986\u76d6\u7684\u914d\u7f6e\u5b57\u5178</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Embed the tokens </p>\n": "<p>\u5d4c\u5165\u4ee3\u5e01</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u751f\u6210\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Run it through the the transformer </p>\n": "<p>\u7528\u5b83\u7a7f\u8fc7\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Run the training loop </p>\n": "<p>\u8fd0\u884c\u8bad\u7ec3\u5faa\u73af</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u6a21\u5757</p>\n",
|
||||
"This is training code with notes for a Fast Weights Transformer.": "\u8fd9\u662f\u5e26\u6709\u5feb\u901f\u6743\u91cd\u53d8\u538b\u5668\u6ce8\u91ca\u7684\u8bad\u7ec3\u4ee3\u7801\u3002",
|
||||
"Train Fast Weights Transformer": "\u8bad\u7ec3\u5feb\u901f\u914d\u91cd\u53d8\u538b\u5668"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">Fast weights transformer</a></h1>\n<p>This is an annotated implementation of the paper <a href=\"https://arxiv.org/abs/2102.11174\">Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch</a>.</p>\n<p>Here is the <a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">annotated implementation</a>. Here are <a href=\"https://nn.labml.ai/transformers/fast_weights/experiment.html\">the training code</a> and a notebook for training a fast weights transformer on the Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30c8\u30e9\u30f3\u30b9</a></h1>\n<p>\u3053\u308c\u306f\u3001PyTorch\u306e\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2102.11174\">\u30ea\u30cb\u30a2\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u5bc6\u304b\u306b\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30e1\u30e2\u30ea\u30b7\u30b9\u30c6\u30e0</a>\u300d\u306b\u6ce8\u91c8\u3092\u4ed8\u3051\u3066\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059\u3002</p>\n<p><a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">\u3053\u308c\u304c\u6ce8\u91c8\u4ed8\u304d\u306e\u5b9f\u88c5\u3067\u3059</a>\u3002\u4ee5\u4e0b\u306f\u3001<a href=\"https://nn.labml.ai/transformers/fast_weights/experiment.html\">Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3068\u30ce\u30fc\u30c8\u30d6\u30c3\u30af\u3067\u3059</a></p>\u3002\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"Fast weights transformer": "\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30c8\u30e9\u30f3\u30b9"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">Fast weights transformer</a></h1>\n<p>This is an annotated implementation of the paper <a href=\"https://arxiv.org/abs/2102.11174\">Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch</a>.</p>\n<p>Here is the <a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">annotated implementation</a>. Here are <a href=\"https://nn.labml.ai/transformers/fast_weights/experiment.html\">the training code</a> and a notebook for training a fast weights transformer on the Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/928aadc0846c11eb85710242ac1c0002\"><span translate=no>_^_1_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">\u0dc0\u0dda\u0d9c\u0dba\u0dd9\u0db1\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0db6\u0dbb</a></h1>\n<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://arxiv.org/abs/2102.11174\">\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0dbb\u0dca\u0da0\u0dca \u0dc4\u0dd2 \u0dbb\u0dc4\u0dc3\u0dd2\u0db1\u0dca \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0db8\u0dad\u0d9a \u0db4\u0daf\u0dca\u0db0\u0dad\u0dd2</a>\u0dc0\u0dda. </p>\n<p>\u0db8\u0dd9\u0db1\u0dca\u0db1 <a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a>. \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad <a href=\"https://nn.labml.ai/transformers/fast_weights/experiment.html\">\u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba\u0dd9\u0dc4\u0dd2 \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dda\u0dad\u0dba</a> \u0dc3\u0dc4 \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0db4\u0ddc\u0dad\u0d9a\u0dca \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/928aadc0846c11eb85710242ac1c0002\"> <span translate=no>_^_1_^_</span></a> </p>\n",
|
||||
"Fast weights transformer": "\u0dc0\u0dda\u0d9c\u0dba\u0dd9\u0db1\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0db6\u0dbb"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">Fast weights transformer</a></h1>\n<p>This is an annotated implementation of the paper <a href=\"https://arxiv.org/abs/2102.11174\">Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch</a>.</p>\n<p>Here is the <a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">annotated implementation</a>. Here are <a href=\"https://nn.labml.ai/transformers/fast_weights/experiment.html\">the training code</a> and a notebook for training a fast weights transformer on the Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">\u5feb\u901f\u79f0\u91cd\u53d8\u538b\u5668</a></h1>\n<p>\u8fd9\u662f <a href=\"https://arxiv.org/abs/2102.11174\">PyTorch \u4e2d\u300a\u7ebf\u6027\u53d8\u538b\u5668\u79d8\u5bc6\u5730\u662f\u5feb\u901f\u52a0\u6743\u5185\u5b58\u7cfb\u7edf\u300b\u4e00\u6587\u7684</a>\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0\u3002</p>\n<p>\u8fd9\u662f<a href=\"https://nn.labml.ai/transformers/fast_weights/index.html\">\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0</a>\u3002\u4ee5\u4e0b\u662f\u7528\u4e8e<a href=\"https://nn.labml.ai/transformers/fast_weights/experiment.html\">\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u5feb\u901f\u6743\u91cd\u8f6c\u6362\u5668\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u548c\u4e00\u672c\u7b14\u8bb0\u672c\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/fast_weights/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"Fast weights transformer": "\u5feb\u901f\u91cd\u91cf\u53d8\u538b\u5668"
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Add the feed-forward results back </p>\n": "<p>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u306e\u7d50\u679c\u3092\u8ffd\u52a0\u3057\u76f4\u3059</p>\n",
|
||||
"<p>Add the self attention results </p>\n": "<p>\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u306e\u7d50\u679c\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Concatenate multiple heads </p>\n": "<p>\u8907\u6570\u306e\u30d8\u30c3\u30c9\u3092\u9023\u7d50</p>\n",
|
||||
"<p>Dropout </p>\n": "<p>\u30c9\u30ed\u30c3\u30d7\u30a2\u30a6\u30c8</p>\n",
|
||||
"<p>Final normalization layer </p>\n": "<p>\u6700\u7d42\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>For each input step </p>\n": "<p>\u5404\u30a4\u30f3\u30d7\u30c3\u30c8\u30b9\u30c6\u30c3\u30d7\u306b\u3064\u3044\u3066</p>\n",
|
||||
"<p>Get layer output </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u51fa\u529b\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>List to store the outputs </p>\n": "<p>\u51fa\u529b\u3092\u4fdd\u5b58\u3059\u308b\u30ea\u30b9\u30c8</p>\n",
|
||||
"<p>Make copies of the transformer layer </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30ec\u30a4\u30e4\u30fc\u306e\u30b3\u30d4\u30fc\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Normalization layers </p>\n": "<p>\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Normalize for feed-forward </p>\n": "<p>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u7528\u306b\u6b63\u898f\u5316</p>\n",
|
||||
"<p>Normalize the output </p>\n": "<p>\u51fa\u529b\u3092\u6b63\u898f\u5316</p>\n",
|
||||
"<p>Number of features per head </p>\n": "<p>\u30d8\u30c3\u30c9\u3042\u305f\u308a\u306e\u6a5f\u80fd\u6570</p>\n",
|
||||
"<p>Output layer </p>\n": "<p>\u51fa\u529b\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Pass through the feed-forward network </p>\n": "<p>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u901a\u904e</p>\n",
|
||||
"<p>Run through each layer </p>\n": "<p>\u5404\u30ec\u30a4\u30e4\u30fc\u3092\u8cab\u901a\u3059\u308b</p>\n",
|
||||
"<p>Split the input to a list along the sequence axis </p>\n": "<p>\u5165\u529b\u3092\u30b7\u30fc\u30b1\u30f3\u30b9\u8ef8\u306b\u6cbf\u3063\u3066\u30ea\u30b9\u30c8\u306b\u5206\u5272\u3057\u307e\u3059</p>\n",
|
||||
"<p>Stack the output tensors </p>\n": "<p>\u51fa\u529b\u30c6\u30f3\u30bd\u30eb\u3092\u7a4d\u307f\u91cd\u306d\u308b</p>\n",
|
||||
"<p>These transform the <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for multi-headed attention. </p>\n": "<p>\u3053\u308c\u3089\u306f\u982d\u306e\u4e2d\u3092\u4e00\u5909\u3055\u305b<span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u591a\u9762\u7684\u306a\u6ce8\u76ee\u3092\u96c6\u3081\u307e\u3059\u3002</p>\n",
|
||||
"<p>These transform the <span translate=no>_^_0_^_</span> multi-headed attention. </p>\n": "<p><span translate=no>_^_0_^_</span>\u3053\u308c\u3089\u306f\u591a\u9762\u7684\u306a\u6ce8\u610f\u529b\u3092\u5909\u3048\u307e\u3059\u3002</p>\n",
|
||||
"<p>Transformer size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5909\u5727\u5668\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"Fast Weight Systems": "\u30d5\u30a1\u30b9\u30c8\u30fb\u30a6\u30a7\u30a4\u30c8\u30fb\u30b7\u30b9\u30c6\u30e0",
|
||||
"This is an annotated implementation/tutorial of Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch.": "\u3053\u308c\u306f\u3001PyTorch\u306e\u30ea\u30cb\u30a2\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u3072\u305d\u304b\u306b\u9ad8\u901f\u30a6\u30a7\u30a4\u30c8\u30e1\u30e2\u30ea\u30b7\u30b9\u30c6\u30e0\u3067\u3042\u308b\u3068\u3044\u3046\u6ce8\u91c8\u4ed8\u304d\u306e\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Add the feed-forward results back </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab\u0d89\u0daf\u0dd2\u0dbb\u0dd2 \u0db4\u0dca\u0dbb\u0dad\u0dd2 results \u0dbd \u0db1\u0dd0\u0dc0\u0dad \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add the self attention results </p>\n": "<p>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db5\u0dbd \u0d91\u0d9a\u0dad\u0dd4 </p>\n",
|
||||
"<p>Concatenate multiple heads </p>\n": "<p>\u0db6\u0dc4\u0dd4\u0dc4\u0dd2\u0dc3\u0dca \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Dropout </p>\n": "<p>\u0dc4\u0dd0\u0dbd\u0dd3\u0db8 </p>\n",
|
||||
"<p>Final normalization layer </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab \u0dc3\u0dca\u0dad\u0dbb\u0dba </p>\n",
|
||||
"<p>For each input step </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d86\u0daf\u0dcf\u0db1 \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dc3\u0db3\u0dc4\u0dcf </p>\n",
|
||||
"<p>Get layer output </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dbb\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>List to store the outputs </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca\u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0 </p>\n",
|
||||
"<p>Make copies of the transformer layer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db4\u0dd2\u0da7\u0db4\u0dad\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Normalization layers </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0dc3\u0dca\u0dae\u0dbb </p>\n",
|
||||
"<p>Normalize for feed-forward </p>\n": "<p>\u0db4\u0ddd\u0dc2\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Normalize the output </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Number of features per head </p>\n": "<p>\u0dc4\u0dd2\u0dc3\u0d9a\u0da7\u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Output layer </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Pass through the feed-forward network </p>\n": "<p>Feed-forward\u0da2\u0dcf\u0dbd\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0d9c\u0db8\u0db1\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run through each layer </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Split the input to a list along the sequence axis </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d85\u0d9a\u0dca\u0dc2\u0dba \u0daf\u0dd2\u0d9c\u0dda \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0da7 \u0d86\u0daf\u0dcf\u0db1\u0dba \u0db6\u0dd9\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Stack the output tensors </p>\n": "<p>\u0db1\u0dd2\u0db8\u0dd0\u0dc0\u0dd4\u0db8\u0dca\u0d86\u0dad\u0dad\u0dd3\u0db1\u0dca \u0d9c\u0ddc\u0da9\u0d9c\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>These transform the <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for multi-headed attention. </p>\n": "<p>\u0db8\u0dda\u0dc0\u0dcf\u0db6\u0dc4\u0dd4 \u0dc1\u0dd3\u0dbb\u0dca\u0dc2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba <span translate=no>_^_1_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2. <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>These transform the <span translate=no>_^_0_^_</span> multi-headed attention. </p>\n": "<p>\u0db8\u0dda\u0dc0\u0dcf <span translate=no>_^_0_^_</span> \u0db6\u0dc4\u0dd4-\u0dc1\u0dd3\u0dbb\u0dca\u0dc2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>Transformer size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"Fast Weight Systems": "\u0dc0\u0dda\u0d9c\u0dba\u0dd9\u0db1\u0dca \u0dc3\u0dd2\u0dbb\u0dd4\u0dbb\u0dda \u0db6\u0dbb \u0db4\u0daf\u0dca\u0db0\u0dad\u0dd2",
|
||||
"This is an annotated implementation/tutorial of Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch.": "\u0db8\u0dd9\u0dba \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0dc4\u0dd2 \u0dbb\u0dc4\u0dc3\u0dd2\u0db1\u0dca \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0db6\u0dbb \u0db8\u0dad\u0d9a \u0db4\u0daf\u0dca\u0db0\u0dad\u0dd2 \u0dc0\u0dda."
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Add the feed-forward results back </p>\n": "<p>\u5c06\u524d\u9988\u7ed3\u679c\u6dfb\u52a0\u56de\u6765</p>\n",
|
||||
"<p>Add the self attention results </p>\n": "<p>\u6dfb\u52a0\u81ea\u6211\u5173\u6ce8\u7684\u7ed3\u679c</p>\n",
|
||||
"<p>Concatenate multiple heads </p>\n": "<p>\u8fde\u63a5\u591a\u4e2a\u5934</p>\n",
|
||||
"<p>Dropout </p>\n": "<p>\u8f8d\u5b66</p>\n",
|
||||
"<p>Final normalization layer </p>\n": "<p>\u6700\u7ec8\u5f52\u4e00\u5316\u5c42</p>\n",
|
||||
"<p>For each input step </p>\n": "<p>\u5bf9\u4e8e\u6bcf\u4e2a\u8f93\u5165\u6b65\u9aa4</p>\n",
|
||||
"<p>Get layer output </p>\n": "<p>\u83b7\u53d6\u56fe\u5c42\u8f93\u51fa</p>\n",
|
||||
"<p>List to store the outputs </p>\n": "<p>\u5b58\u50a8\u8f93\u51fa\u7684\u5217\u8868</p>\n",
|
||||
"<p>Make copies of the transformer layer </p>\n": "<p>\u5236\u4f5c\u53d8\u538b\u5668\u5c42\u7684\u526f\u672c</p>\n",
|
||||
"<p>Normalization layers </p>\n": "<p>\u5f52\u4e00\u5316\u5c42</p>\n",
|
||||
"<p>Normalize for feed-forward </p>\n": "<p>\u6807\u51c6\u5316\u4ee5\u8fdb\u884c\u524d\u9988</p>\n",
|
||||
"<p>Normalize the output </p>\n": "<p>\u89c4\u8303\u5316\u8f93\u51fa</p>\n",
|
||||
"<p>Number of features per head </p>\n": "<p>\u6bcf\u5934\u7279\u5f81\u6570</p>\n",
|
||||
"<p>Output layer </p>\n": "<p>\u8f93\u51fa\u5c42</p>\n",
|
||||
"<p>Pass through the feed-forward network </p>\n": "<p>\u901a\u8fc7\u524d\u9988\u7f51\u7edc</p>\n",
|
||||
"<p>Run through each layer </p>\n": "<p>\u7a7f\u8fc7\u6bcf\u4e00\u5c42</p>\n",
|
||||
"<p>Split the input to a list along the sequence axis </p>\n": "<p>\u6cbf\u5e8f\u5217\u8f74\u5c06\u8f93\u5165\u62c6\u5206\u4e3a\u4e00\u4e2a\u5217\u8868</p>\n",
|
||||
"<p>Stack the output tensors </p>\n": "<p>\u5806\u53e0\u8f93\u51fa\u5f20\u91cf</p>\n",
|
||||
"<p>These transform the <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for multi-headed attention. </p>\n": "<p>\u8fd9\u4e9b\u6539\u53d8\u4e86\u591a<span translate=no>_^_1_^_</span>\u5934\u6ce8\u610f\u529b\u7684<span translate=no>_^_0_^_</span>\u548c\u3002</p>\n",
|
||||
"<p>These transform the <span translate=no>_^_0_^_</span> multi-headed attention. </p>\n": "<p>\u8fd9\u4e9b\u6539\u53d8\u4e86<span translate=no>_^_0_^_</span>\u591a\u5934\u6ce8\u610f\u529b\u3002</p>\n",
|
||||
"<p>Transformer size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u53d8\u538b\u5668\u5c3a\u5bf8<span translate=no>_^_0_^_</span></p>\n",
|
||||
"Fast Weight Systems": "\u5feb\u901f\u79f0\u91cd\u7cfb\u7edf",
|
||||
"This is an annotated implementation/tutorial of Linear Transformers Are Secretly Fast Weight Memory Systems in PyTorch.": "\u8fd9\u662f PyTorch \u4e2d\u7ebf\u6027\u53d8\u538b\u5668\u662f\u79d8\u5bc6\u7684\u5feb\u901f\u91cd\u91cf\u5b58\u50a8\u7cfb\u7edf\u7684\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0/\u6559\u7a0b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"<h1>Position-wise Feed-Forward Network (FFN)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of position-wise feedforward network used in transformer.</p>\n<p>FFN consists of two fully connected layers. Number of dimensions in the hidden layer <span translate=no>_^_0_^_</span>, is generally set to around four times that of the token embedding <span translate=no>_^_1_^_</span>. So it is sometime also called the expand-and-contract network.</p>\n<p>There is an activation at the hidden layer, which is usually set to ReLU (Rectified Linear Unit) activation, <span translate=no>_^_2_^_</span></p>\n<p>That is, the FFN function is, <span translate=no>_^_3_^_</span> where <span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>, <span translate=no>_^_6_^_</span> and <span translate=no>_^_7_^_</span> are learnable parameters.</p>\n<p>Sometimes the GELU (Gaussian Error Linear Unit) activation is also used instead of ReLU. <span translate=no>_^_8_^_</span> where <span translate=no>_^_9_^_</span></p>\n<h3>Gated Linear Units</h3>\n<p>This is a generic implementation that supports different variants including <a href=\"https://arxiv.org/abs/2002.05202\">Gated Linear Units</a> (GLU). We have also implemented experiments on these:</p>\n<ul><li><a href=\"glu_variants/experiment.html\">experiment that uses <span translate=no>_^_10_^_</span></a> </li>\n<li><a href=\"glu_variants/simple.html\">simpler version from scratch</a></li></ul>\n": "<h1>\u4f4d\u7f6e\u5225\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af (FFN)</h1>\n<p>\u3053\u308c\u306f\u3001<a href=\"https://pytorch.org\">\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3067\u4f7f\u7528\u3055\u308c\u308b\u4f4d\u7f6e\u5358\u4f4d\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306ePyTorch\u5b9f\u88c5\u3067\u3059</a>\u3002</p>\n<p>FFN \u306f\u3001\u5b8c\u5168\u306b\u63a5\u7d9a\u3055\u308c\u305f 2 \u3064\u306e\u30ec\u30a4\u30e4\u30fc\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u96a0\u3057\u30ec\u30a4\u30e4\u30fc\u306e\u6b21\u5143\u6570\u306f<span translate=no>_^_0_^_</span>\u3001\u901a\u5e38\u3001<span translate=no>_^_1_^_</span>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u306e\u7d044\u500d\u306b\u8a2d\u5b9a\u3055\u308c\u307e\u3059\u3002\u305d\u306e\u305f\u3081\u3001\u30a8\u30af\u30b9\u30d1\u30f3\u30c9\u30fb\u30b3\u30f3\u30c8\u30e9\u30af\u30c8\u30fb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3068\u547c\u3070\u308c\u308b\u3053\u3068\u3082\u3042\u308a\u307e\u3059</p>\u3002\n<p>\u96a0\u308c\u5c64\u306b\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u304c\u3042\u308a\u3001\u901a\u5e38\u306fReLU (Rectified Linear Unit) \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306b\u8a2d\u5b9a\u3055\u308c\u3066\u3044\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n<p>\u3064\u307e\u308a\u3001FFN \u95a2\u6570\u306f\u3001\u3001\u3001\u3001<span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span><span translate=no>_^_6_^_</span>\u3001<span translate=no>_^_3_^_</span><span translate=no>_^_7_^_</span>\u306f\u5b66\u7fd2\u53ef\u80fd\u306a\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3067\u3059\u3002</p>\n<p>ReLU \u306e\u4ee3\u308f\u308a\u306b GELU (\u30ac\u30a6\u30b9\u8aa4\u5dee\u7dda\u5f62\u5358\u4f4d) \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u304c\u4f7f\u7528\u3055\u308c\u308b\u3053\u3068\u3082\u3042\u308a\u307e\u3059\u3002<span translate=no>_^_8_^_</span>\u3069\u3053 <span translate=no>_^_9_^_</span></p>\n<h3>\u30b2\u30fc\u30c8\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8</h3>\n<p>\u3053\u308c\u306f\u3001<a href=\"https://arxiv.org/abs/2002.05202\">\u30b2\u30fc\u30c8\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8</a> (GLU) \u3092\u542b\u3080\u3055\u307e\u3056\u307e\u306a\u30d0\u30ea\u30a2\u30f3\u30c8\u3092\u30b5\u30dd\u30fc\u30c8\u3059\u308b\u4e00\u822c\u7684\u306a\u5b9f\u88c5\u3067\u3059\u3002\u307e\u305f\u3001\u4ee5\u4e0b\u306e\u5b9f\u9a13\u3082\u884c\u3063\u3066\u3044\u307e\u3059</p>\u3002\n<ul><li><a href=\"glu_variants/experiment.html\">\u3092\u4f7f\u7528\u3059\u308b\u5b9f\u9a13 <span translate=no>_^_10_^_</span></a></li>\n<li><a href=\"glu_variants/simple.html\">\u30bc\u30ed\u304b\u3089\u306e\u30b7\u30f3\u30d7\u30eb\u306a\u30d0\u30fc\u30b8\u30e7\u30f3</a></li></ul>\n",
|
||||
"<h2>FFN module</h2>\n": "<h2>FFN \u30e2\u30b8\u30e5\u30fc\u30eb</h2>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> or <span translate=no>_^_1_^_</span> depending on whether it is gated </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u30b2\u30fc\u30c8\u4ed8\u304d\u304b\u5426\u304b\u306b\u3088\u308b\u3051\u3069</p>\n",
|
||||
"<p>Activation function <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u6a5f\u80fd <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Apply dropout </p>\n": "<p>\u30c9\u30ed\u30c3\u30d7\u30a2\u30a6\u30c8\u3092\u9069\u7528</p>\n",
|
||||
"<p>Hidden layer dropout </p>\n": "<p>\u96a0\u3057\u30ec\u30a4\u30e4\u30fc\u306e\u30c9\u30ed\u30c3\u30d7\u30a2\u30a6\u30c8</p>\n",
|
||||
"<p>If gated, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b2\u30fc\u30c8\u3055\u308c\u3066\u3044\u308b\u5834\u5408\u3001<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>If there is a gate the linear layer to transform inputs to be multiplied by the gate, parameterized by weight <span translate=no>_^_0_^_</span> and bias <span translate=no>_^_1_^_</span> </p>\n": "<p>\u30b2\u30fc\u30c8\u304c\u3042\u308b\u5834\u5408\u306f\u3001\u5165\u529b\u3092\u5909\u63db\u3057\u3066\u30b2\u30fc\u30c8\u3092\u639b\u3051\u3001\u30a6\u30a7\u30a4\u30c8\u3068\u30d0\u30a4\u30a2\u30b9\u3092\u30d1\u30e9\u30e1\u30fc\u30bf\u5316\u3057\u3066\u5165\u529b\u3092\u5909\u63db\u3059\u308b\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Layer one parameterized by weight <span translate=no>_^_0_^_</span> and bias <span translate=no>_^_1_^_</span> </p>\n": "<p>\u91cd\u307f\u3068\u30d0\u30a4\u30a2\u30b9\u3067\u30d1\u30e9\u30e1\u30fc\u30bf\u5316\u3055\u308c\u305f\u30ec\u30a4\u30e4\u30fc 1 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Otherwise </p>\n": "<p>\u305d\u308c\u4ee5\u5916\u306e\u5834\u5408</p>\n",
|
||||
"<p>Whether there is a gate </p>\n": "<p>\u30b2\u30fc\u30c8\u304c\u3042\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in a token embedding </li>\n<li><span translate=no>_^_1_^_</span> is the number of features in the hidden layer of the FFN </li>\n<li><span translate=no>_^_2_^_</span> is dropout probability for the hidden layer </li>\n<li><span translate=no>_^_3_^_</span> specifies whether the hidden layer is gated </li>\n<li><span translate=no>_^_4_^_</span> specified whether the first fully connected layer should have a learnable bias </li>\n<li><span translate=no>_^_5_^_</span> specified whether the second fully connected layer should have a learnable bias </li>\n<li><span translate=no>_^_6_^_</span> specified whether the fully connected layer for the gate should have a learnable bias</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u306b\u542b\u307e\u308c\u308b\u6a5f\u80fd\u306e\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u306f FFN \u306e\u96a0\u308c\u30ec\u30a4\u30e4\u30fc\u306b\u3042\u308b\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u96a0\u308c\u5c64\u306e\u30c9\u30ed\u30c3\u30d7\u30a2\u30a6\u30c8\u78ba\u7387\u3067\u3059</li>\n<li><span translate=no>_^_3_^_</span>\u96a0\u308c\u30ec\u30a4\u30e4\u30fc\u3092\u30b2\u30fc\u30c8\u3059\u308b\u304b\u3069\u3046\u304b\u3092\u6307\u5b9a\u3057\u307e\u3059</li>\n<li><span translate=no>_^_4_^_</span>\u6700\u521d\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u4ed8\u3051\u308b\u304b\u3069\u3046\u304b\u3092\u6307\u5b9a\u3057\u307e\u3057\u305f</li>\n<li><span translate=no>_^_5_^_</span>2 \u756a\u76ee\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u4ed8\u3051\u308b\u304b\u3069\u3046\u304b\u3092\u6307\u5b9a\u3057\u307e\u3057\u305f</li>\n<li><span translate=no>_^_6_^_</span>\u30b2\u30fc\u30c8\u306e\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u8a2d\u3051\u308b\u3079\u304d\u304b\u3069\u3046\u304b\u3092\u6307\u5b9a</li></ul>\n",
|
||||
"Documented reusable implementation of the position wise feedforward network.": "\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u518d\u5229\u7528\u53ef\u80fd\u306a\u5b9f\u88c5\u304c\u6587\u66f8\u5316\u3055\u308c\u307e\u3057\u305f\u3002",
|
||||
"Position-wise Feed-Forward Network (FFN)": "\u4f4d\u7f6e\u5225\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af (FFN)"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"<h1>Position-wise Feed-Forward Network (FFN)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of position-wise feedforward network used in transformer.</p>\n<p>FFN consists of two fully connected layers. Number of dimensions in the hidden layer <span translate=no>_^_0_^_</span>, is generally set to around four times that of the token embedding <span translate=no>_^_1_^_</span>. So it is sometime also called the expand-and-contract network.</p>\n<p>There is an activation at the hidden layer, which is usually set to ReLU (Rectified Linear Unit) activation, <span translate=no>_^_2_^_</span></p>\n<p>That is, the FFN function is, <span translate=no>_^_3_^_</span> where <span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>, <span translate=no>_^_6_^_</span> and <span translate=no>_^_7_^_</span> are learnable parameters.</p>\n<p>Sometimes the GELU (Gaussian Error Linear Unit) activation is also used instead of ReLU. <span translate=no>_^_8_^_</span> where <span translate=no>_^_9_^_</span></p>\n<h3>Gated Linear Units</h3>\n<p>This is a generic implementation that supports different variants including <a href=\"https://arxiv.org/abs/2002.05202\">Gated Linear Units</a> (GLU). We have also implemented experiments on these:</p>\n<ul><li><a href=\"glu_variants/experiment.html\">experiment that uses <span translate=no>_^_10_^_</span></a> </li>\n<li><a href=\"glu_variants/simple.html\">simpler version from scratch</a></li></ul>\n": "<h1>\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc \uff08FFN\uff09</h1>\n<p>\u8fd9\u662f Transformer \u4e2d\u4f7f\u7528\u7684\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc\u7684 <a href=\"https://pytorch.org\"> PyTorch </a> \u5b9e\u73b0\u3002</p>\n<p> FFN \u7531\u4e24\u4e2a\u5168\u8fde\u63a5\u5c42\u7ec4\u6210\u3002\u9690\u85cf\u5c42\u4e2d\u7684\u7ef4\u5ea6\u6570<span translate=no>_%5e_0_%5e_</span>\u901a\u5e38\u8bbe\u7f6e\u4e3a\u6807\u8bb0\u5d4c\u5165\u7ef4\u5ea6<span translate=no>_%5e_1_%5e_</span>\u7684\u56db\u500d\u5de6\u53f3\u3002\u56e0\u6b64\uff0c\u5b83\u6709\u65f6\u4e5f\u88ab\u79f0\u4e3a\u6269\u5f20-\u538b\u7f29\u7f51\u7edc\u3002</p>\n<p>\u9690\u85cf\u5c42\u6709\u4e00\u4e2a\u6fc0\u6d3b\u51fd\u6570\uff0c\u901a\u5e38\u8bbe\u7f6e\u4e3a ReLU (Rectified Linear Unit) \u6fc0\u6d3b\u51fd\u6570\uff0c<span translate=no>_%5e_2_%5e_</span></p>\n<p>\u5728\u6b64\u57fa\u7840\u4e0a\uff0c FFN \u51fd\u6570\u53ef\u4ee5\u5199\u4f5c\uff1a<span translate=no>_%5e_3_%5e_</span>\u5176\u4e2d<span translate=no>_%5e_4_%5e_</span><span translate=no>_%5e_5_%5e_</span>\u3001<span translate=no>_%5e_6_%5e_</span>\u548c<span translate=no>_%5e_7_%5e_</span>\u662f\u53ef\u5b66\u4e60\u7684\u53c2\u6570\u3002</p>\n<p>\u6709\u65f6\u8fd8\u4f1a\u4f7f\u7528 GELU (Gaussian Error Linear Unit) \u6fc0\u6d3b\u51fd\u6570\u6765\u4ee3\u66ff ReLU \u3002<span translate=no>_%5e_8_%5e_</span>\u5176\u4e2d<span translate=no>_%5e_9_%5e_</span></p>\n<h3>\u95e8\u63a7\u7ebf\u6027\u5355\u5143</h3>\n<p>\u8fd9\u662f\u4e00\u4e2a\u901a\u7528\u5b9e\u73b0\uff0c\u652f\u6301\u5305\u62ec<a href=\"https://arxiv.org/abs/2002.05202\">\u95e8\u63a7\u7ebf\u6027\u5355\u5143(GLU)</a> \u5728\u5185\u7684\u4e0d\u540c\u53d8\u4f53\u3002\u6211\u4eec\u8fd8\u5bf9\u8fd9\u4e9b\u8fdb\u884c\u4e86\u5b9e\u9a8c\uff1a</p>\n<ul><li><a href=\"glu_variants/experiment.html\">\u4f7f\u7528<span translate=no>_%5e_10_%5e_</span></a>\u7684\u5b9e\u9a8c</li>\n<li><a href=\"glu_variants/simple.html\">\u4ece\u5934\u5f00\u59cb\u7684\u7b80\u5316\u7248\u672c</a></li></ul>\n",
|
||||
"<h2>FFN module</h2>\n": "<h2>FFN \u6a21\u5757</h2>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> or <span translate=no>_^_1_^_</span> depending on whether it is gated </p>\n": "<p>\u6839\u636e\u662f\u5426\u8fdb\u884c\u95e8\u63a7\uff0c\u8fd4\u56de<span translate=no>_^_0_^_</span>\u6216\u8005<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Activation function <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6fc0\u6d3b\u51fd\u6570<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Apply dropout </p>\n": "<p>\u4f7f\u7528 Dropout</p>\n",
|
||||
"<p>Hidden layer dropout </p>\n": "<p>\u9690\u85cf\u5c42 Dropout</p>\n",
|
||||
"<p>If gated, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u8fdb\u884c\u95e8\u63a7\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>If there is a gate the linear layer to transform inputs to be multiplied by the gate, parameterized by weight <span translate=no>_^_0_^_</span> and bias <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5982\u679c\u5b58\u5728\u95e8\u63a7\uff0c\u5219\u901a\u8fc7\u7ebf\u6027\u5c42\u5c06\u8f93\u5165\u503c\u4e0e\u95e8\u76f8\u4e58\uff0c\u5e76\u7531\u6743\u91cd <span translate=no>_^_0_^_</span>\u548c\u504f\u7f6e<span translate=no>_^_1_^_</span>\u8fdb\u884c\u53c2\u6570\u5316</p>\n",
|
||||
"<p>Layer one parameterized by weight <span translate=no>_^_0_^_</span> and bias <span translate=no>_^_1_^_</span> </p>\n": "<p>\u7b2c\u4e00\u5c42\u7531\u6743\u91cd<span translate=no>_^_0_^_</span>\u548c\u504f\u5dee<span translate=no>_^_1_^_</span>\u8fdb\u884c\u53c2\u6570\u5316</p>\n",
|
||||
"<p>Otherwise </p>\n": "<p>\u5426\u5219</p>\n",
|
||||
"<p>Whether there is a gate </p>\n": "<p>\u662f\u5426\u5b58\u5728\u95e8\u63a7</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in a token embedding </li>\n<li><span translate=no>_^_1_^_</span> is the number of features in the hidden layer of the FFN </li>\n<li><span translate=no>_^_2_^_</span> is dropout probability for the hidden layer </li>\n<li><span translate=no>_^_3_^_</span> specifies whether the hidden layer is gated </li>\n<li><span translate=no>_^_4_^_</span> specified whether the first fully connected layer should have a learnable bias </li>\n<li><span translate=no>_^_5_^_</span> specified whether the second fully connected layer should have a learnable bias </li>\n<li><span translate=no>_^_6_^_</span> specified whether the fully connected layer for the gate should have a learnable bias</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u6807\u8bb0\u5d4c\u5165\u4e2d\u7684\u7279\u5f81\u6570\u91cf</li>\n<li><span translate=no>_^_1_^_</span>\u662f FFN \u9690\u85cf\u5c42\u4e2d\u7684\u7279\u5f81\u6570\u91cf</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u9690\u85cf\u5c42\u7684 Dropout \u7387</li>\n<li><span translate=no>_^_3_^_</span>\u6307\u5b9a\u4e86\u9690\u85cf\u5c42\u662f\u5426\u4e3a\u95e8\u63a7\u5c42</li>\n<li><span translate=no>_^_4_^_</span>\u6307\u5b9a\u4e86\u7b2c\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5e94\u8be5\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e</li>\n<li><span translate=no>_^_5_^_</span>\u6307\u5b9a\u7b2c\u4e8c\u4e2a\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5e94\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e</li>\n<li><span translate=no>_^_6_^_</span>\u6307\u5b9a\u95e8\u63a7\u7684\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5e94\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e</li></ul>\n",
|
||||
"Documented reusable implementation of the position wise feedforward network.": "\u5df2\u8bb0\u5f55\u5e76\u53ef\u91cd\u590d\u4f7f\u7528\u7684\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc\u5b9e\u73b0\u3002",
|
||||
"Position-wise Feed-Forward Network (FFN)": "\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc (FFN)"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>Train Feedback Transformer</h1>\n<p>This trains a <a href=\"index.html\">feedback transformer</a> model for auto-regression. You can pick the original feedback transformer or the new version where the keys and values are precalculated.</p>\n<p>Here's a Colab notebook for training a feedback transformer on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/feedback/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u30c8\u30ec\u30a4\u30f3\u30fb\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30fb\u30c8\u30e9\u30f3\u30b9</h1>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001<a href=\"index.html\">\u81ea\u5df1\u56de\u5e30\u7528\u306e\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u304c\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3055\u308c\u307e\u3059</a>\u3002\u30aa\u30ea\u30b8\u30ca\u30eb\u306e\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u304b\u3001\u30ad\u30fc\u3068\u5024\u304c\u4e8b\u524d\u306b\u8a08\u7b97\u3055\u308c\u305f\u65b0\u3057\u3044\u30d0\u30fc\u30b8\u30e7\u30f3\u3092\u9078\u629e\u3067\u304d\u307e\u3059</p>\u3002\n<p>\u3053\u308c\u306f\u3001Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306eColab\u30ce\u30fc\u30c8\u30d6\u30c3\u30af\u3067\u3059\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/feedback/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52d5\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u8a2d\u5b9a\u306f\u3001\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u305f\u3068\u304d\u306b\u4e0a\u66f8\u304d\u3067\u304d\u3001\u307e\u305f\u4e0a\u66f8\u304d\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p> Create <a href=\"index.html\">original feedback transformer</a>.</p>\n": "<p><a href=\"index.html\">\u30aa\u30ea\u30b8\u30ca\u30eb\u306e\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u4f5c\u6210</a></p>\n",
|
||||
"<p> Create <a href=\"index.html#kv_shared\">updated feedback transformer</a>, with precalculated keys and values.</p>\n": "<p>\u4e8b\u524d\u306b\u8a08\u7b97\u3055\u308c\u305f\u30ad\u30fc\u3068\u5024\u3092\u4f7f\u7528\u3057\u3066\u3001<a href=\"index.html#kv_shared\">\u66f4\u65b0\u3055\u308c\u305f\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u4f5c\u6210\u3057\u307e\u3059</a>\u3002</p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u3059\u308b\u8a2d\u5b9a\u306e\u8f9e\u66f8</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Embed the tokens </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u3092\u57cb\u3081\u8fbc\u3080</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Run it through the the transformer </p>\n": "<p>\u5909\u5727\u5668\u306b\u901a\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Run the training loop </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3092\u5b9f\u884c\u3059\u308b</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"<p>Use <span translate=no>_^_0_^_</span> for original feedback transformer </p>\n": "<p><span translate=no>_^_0_^_</span>\u30aa\u30ea\u30b8\u30ca\u30eb\u306e\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30c8\u30e9\u30f3\u30b9\u306b\u4f7f\u7528</p>\n",
|
||||
"This is training code with notes for a feedback transformer.": "\u3053\u308c\u306f\u3001\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306e\u30e1\u30e2\u3092\u542b\u3080\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3067\u3059\u3002",
|
||||
"Train Feedback Transformer": "\u30c8\u30ec\u30a4\u30f3\u30fb\u30d5\u30a3\u30fc\u30c9\u30d0\u30c3\u30af\u30fb\u30c8\u30e9\u30f3\u30b9"
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>Train Feedback Transformer</h1>\n<p>This trains a <a href=\"index.html\">feedback transformer</a> model for auto-regression. You can pick the original feedback transformer or the new version where the keys and values are precalculated.</p>\n<p>Here's a Colab notebook for training a feedback transformer on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/feedback/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/d8eb9416530a11eb8fb50242ac1c0002\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</h1>\n<p>\u0db8\u0dd9\u0dba\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0dc0 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf <a href=\"index.html\">\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a> \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2. \u0dba\u0dad\u0dd4\u0dbb\u0dd4 \u0dc3\u0dc4 \u0d85\u0d9c\u0dba\u0db1\u0dca \u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0db6\u0db1 \u0db8\u0dd4\u0dbd\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba \u0dc4\u0ddd \u0db1\u0dc0 \u0d85\u0db1\u0dd4\u0dc0\u0dcf\u0daf\u0dba \u0d94\u0db6\u0da7 \u0dad\u0ddd\u0dbb\u0dcf \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. </p>\n<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0ddc\u0dbd\u0db6\u0dca \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0db4\u0ddc\u0dad\u0d9a\u0dca \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/feedback/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/d8eb9416530a11eb8fb50242ac1c0002\"> <span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0d85\u0db4\u0dd2\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1 \u0dc0\u0dd2\u0da7 \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3 \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0d85\u0dad\u0dbb \u0d91\u0dba \u0d85\u0db0\u0dd2\u0d9a \u0dbd\u0dd9\u0dc3 \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0dc0\u0db1\u0dd4 \u0d87\u0dad</p>\n",
|
||||
"<p> Create <a href=\"index.html\">original feedback transformer</a>.</p>\n": "<p> <a href=\"index.html\">\u0db8\u0dd4\u0dbd\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba</a>\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p> Create <a href=\"index.html#kv_shared\">updated feedback transformer</a>, with precalculated keys and values.</p>\n": "<p> \u0db4\u0dd6\u0dbb\u0dca\u0dc0\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dc5 \u0dba\u0dad\u0dd4\u0dbb\u0dd4 \u0dc3\u0dc4 \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc3\u0db8\u0d9f <a href=\"index.html#kv_shared\">\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca</a>\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf\u0dba\u0dcf\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc1\u0db6\u0dca\u0daf\u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dca </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Embed the tokens </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 </p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run it through the the transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0dc4\u0dbb\u0dc4\u0dcf \u0d91\u0dba \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run the training loop </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dbd\u0dd6\u0db4\u0dba \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"<p>Use <span translate=no>_^_0_^_</span> for original feedback transformer </p>\n": "<p>\u0db8\u0dd4\u0dbd\u0dca\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <span translate=no>_^_0_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"This is training code with notes for a feedback transformer.": "\u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0dc3\u0dc4\u0dd2\u0dad \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dda\u0dad\u0dba\u0d9a\u0dd2.",
|
||||
"Train Feedback Transformer": "\u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0ddd\u0dc2\u0dab \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca"
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>Train Feedback Transformer</h1>\n<p>This trains a <a href=\"index.html\">feedback transformer</a> model for auto-regression. You can pick the original feedback transformer or the new version where the keys and values are precalculated.</p>\n<p>Here's a Colab notebook for training a feedback transformer on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/feedback/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u706b\u8f66\u53cd\u9988\u53d8\u538b\u5668</h1>\n<p>\u8fd9\u4f1a\u8bad\u7ec3\u4e00\u4e2a\u7528\u4e8e\u81ea\u52a8\u56de\u5f52\u7684<a href=\"index.html\">\u53cd\u9988\u53d8\u538b\u5668</a>\u6a21\u578b\u3002\u60a8\u53ef\u4ee5\u9009\u62e9\u539f\u59cb\u53cd\u9988\u53d8\u538b\u5668\uff0c\u4e5f\u53ef\u4ee5\u9009\u62e9\u9884\u5148\u8ba1\u7b97\u5bc6\u94a5\u548c\u503c\u7684\u65b0\u7248\u672c\u3002</p>\n<p>\u8fd9\u662f\u4e00\u672c Colab \u7b14\u8bb0\u672c\uff0c\u7528\u4e8e\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u53cd\u9988\u8f6c\u6362\u5668\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/feedback/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52a8\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u5f53\u6211\u4eec\u5f00\u59cb\u5b9e\u9a8c\u65f6\uff0c\u9ed8\u8ba4\u914d\u7f6e\u53ef\u4ee5\u800c\u4e14\u5c06\u4f1a\u88ab\u8986\u76d6</p>\n",
|
||||
"<p> Create <a href=\"index.html\">original feedback transformer</a>.</p>\n": "<p>\u521b\u5efa<a href=\"index.html\">\u539f\u59cb\u7684\u53cd\u9988\u53d8\u538b\u5668</a>\u3002</p>\n",
|
||||
"<p> Create <a href=\"index.html#kv_shared\">updated feedback transformer</a>, with precalculated keys and values.</p>\n": "\u4f7f\u7528@@ <p>\u9884\u5148\u8ba1\u7b97\u7684\u952e\u548c\u503c\u521b\u5efa<a href=\"index.html#kv_shared\">\u66f4\u65b0\u7684\u53cd\u9988\u8f6c\u6362\u5668</a>\u3002</p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u8981\u8986\u76d6\u7684\u914d\u7f6e\u5b57\u5178</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Embed the tokens </p>\n": "<p>\u5d4c\u5165\u4ee3\u5e01</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u751f\u6210\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Run it through the the transformer </p>\n": "<p>\u7528\u5b83\u7a7f\u8fc7\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Run the training loop </p>\n": "<p>\u8fd0\u884c\u8bad\u7ec3\u5faa\u73af</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u6a21\u5757</p>\n",
|
||||
"<p>Use <span translate=no>_^_0_^_</span> for original feedback transformer </p>\n": "<p><span translate=no>_^_0_^_</span>\u7528\u4e8e\u539f\u88c5\u53cd\u9988\u53d8\u538b\u5668</p>\n",
|
||||
"This is training code with notes for a feedback transformer.": "\u8fd9\u662f\u5e26\u6709\u53cd\u9988\u8f6c\u6362\u5668\u6ce8\u91ca\u7684\u8bad\u7ec3\u4ee3\u7801\u3002",
|
||||
"Train Feedback Transformer": "\u5217\u8f66\u53cd\u9988\u53d8\u538b\u5668"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"<h1>FNet: Mixing Tokens with Fourier Transforms</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"../mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a <span translate=no>_^_0_^_</span> more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>.</p>\n<h2>Mixing tokens with two Fourier transforms</h2>\n<p>We apply Fourier transform along the hidden dimension (embedding dimension) and then along the sequence dimension.</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>where <span translate=no>_^_2_^_</span> is the embedding input, <span translate=no>_^_3_^_</span> stands for the fourier transform and <span translate=no>_^_4_^_</span> stands for the real component in complex numbers.</p>\n<p>This is very simple to implement on PyTorch - just 1 line of code. The paper suggests using a precomputed DFT matrix and doing matrix multiplication to get the Fourier transformation.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for using a FNet based model for classifying <a href=\"https://paperswithcode.com/dataset/ag-news\">AG News</a>.</p>\n": "<h1>FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408</h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.03824\">FNet: \u30c8\u30fc\u30af\u30f3\u3092\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3068\u6df7\u5408\u3059\u308b\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n<p><em>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<a href=\"../mha.html\"><a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u5c64\u30922\u3064\u306e\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u7f6e\u304d\u63db\u3048\u3066\u30c8\u30fc\u30af\u30f3\u3092\u6df7\u5408\u3057\u307e\u3059</a></a>\u3002</em><span translate=no>_^_0_^_</span>\u3053\u308c\u306f\u81ea\u5df1\u6ce8\u610f\u3088\u308a\u3082\u52b9\u7387\u7684\u3067\u3059\u3002<a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE</a> \u30d9\u30f3\u30c1\u30de\u30fc\u30af\u3067\u306f\u3001\u81ea\u5df1\u6ce8\u610f\u3088\u308a\u3082\u3053\u308c\u3092\u4f7f\u7528\u3057\u305f\u5834\u5408\u306e\u7cbe\u5ea6\u306e\u4f4e\u4e0b\u306f\u7d04 92%</p> \u3067\u3059\u3002\n<h2>2 \u3064\u306e\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408</h2>\n<p>\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u975e\u8868\u793a\u6b21\u5143 (\u57cb\u3081\u8fbc\u307f\u6b21\u5143) \u306b\u6cbf\u3063\u3066\u9069\u7528\u3057\u3001\u6b21\u306b\u30b7\u30fc\u30b1\u30f3\u30b9\u6b21\u5143\u306b\u6cbf\u3063\u3066\u9069\u7528\u3057\u307e\u3059\u3002</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>\u3053\u3053\u3067\u3001<span translate=no>_^_2_^_</span>\u306f\u57cb\u3081\u8fbc\u307f\u5165\u529b\u3067\u3001<span translate=no>_^_3_^_</span><span translate=no>_^_4_^_</span>\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u8868\u3057\u3001\u8907\u7d20\u6570\u306e\u5b9f\u6570\u6210\u5206\u3092\u8868\u3057\u307e\u3059\u3002</p>\n<p>\u3053\u308c\u3092PyTorch\u306b\u5b9f\u88c5\u3059\u308b\u306e\u306f\u3068\u3066\u3082\u7c21\u5358\u3067\u3059\u3002\u305f\u3063\u305f1\u884c\u306e\u30b3\u30fc\u30c9\u3067\u3059\u3002\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u4e8b\u524d\u306b\u8a08\u7b97\u3055\u308c\u305fDFT\u884c\u5217\u3092\u4f7f\u7528\u3057\u3001\u884c\u5217\u306e\u4e57\u7b97\u3092\u884c\u3063\u3066\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u884c\u3046\u3053\u3068\u3092\u63d0\u6848\u3057\u3066\u3044\u307e\u3059</p>\u3002\n<p>\u4ee5\u4e0b\u306f<a href=\"experiment.html\">\u3001<a href=\"https://paperswithcode.com/dataset/ag-news\">FNet\u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u3092\u4f7f\u7528\u3057\u3066AG</a> News\u3092\u5206\u985e\u3059\u308b\u305f\u3081\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3067\u3059</a>\u3002</p>\n",
|
||||
"<h2>FNet - Mix tokens</h2>\n<p>This module simply implements <span translate=no>_^_0_^_</span></p>\n<p>The structure of this module is made similar to a <a href=\"../mha.html\">standard attention module</a> so that we can simply replace it.</p>\n": "<h2>FNet-\u30df\u30c3\u30af\u30b9\u30c8\u30fc\u30af\u30f3</h2>\n<p>\u3053\u306e\u30e2\u30b8\u30e5\u30fc\u30eb\u306f\u5358\u7d14\u306b\u5b9f\u88c5\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n<p>\u3053\u306e\u30e2\u30b8\u30e5\u30fc\u30eb\u306e\u69cb\u9020\u306f\u3001<a href=\"../mha.html\">\u6a19\u6e96\u7684\u306a\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30b8\u30e5\u30fc\u30eb\u3068\u540c\u69d8\u306e\u69cb\u9020\u306b\u306a\u3063\u3066\u3044\u308b\u305f\u3081</a>\u3001\u7c21\u5358\u306b\u4ea4\u63db\u3067\u304d\u307e\u3059\u3002</p>\n",
|
||||
"<p> The <a href=\"../mha.html\">normal attention module</a> can be fed with different token embeddings for <span translate=no>_^_0_^_</span>,<span translate=no>_^_1_^_</span>, and <span translate=no>_^_2_^_</span> and a mask.</p>\n<p>We follow the same function signature so that we can replace it directly.</p>\n<p>For FNet mixing, <span translate=no>_^_3_^_</span> and masking is not possible. Shape of <span translate=no>_^_4_^_</span> (and <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>) is <span translate=no>_^_7_^_</span>.</p>\n": "<p><a href=\"../mha.html\">\u901a\u5e38\u306e\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30b8\u30e5\u30fc\u30eb\u306b\u306f</a>\u3001\u3001\u3001<span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u30de\u30b9\u30af\u306b\u3055\u307e\u3056\u307e\u306a\u30c8\u30fc\u30af\u30f3\u3092\u57cb\u3081\u8fbc\u3080\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n<p>\u540c\u3058\u95a2\u6570\u30b7\u30b0\u30cd\u30c1\u30e3\u306b\u5f93\u3046\u306e\u3067\u3001\u76f4\u63a5\u7f6e\u63db\u3067\u304d\u307e\u3059\u3002</p>\n<p>FNet\u30df\u30ad\u30b7\u30f3\u30b0\u306e\u5834\u5408<span translate=no>_^_3_^_</span>\u3001\u30de\u30b9\u30ad\u30f3\u30b0\u306f\u3067\u304d\u307e\u305b\u3093\u3002<span translate=no>_^_4_^_</span>(<span translate=no>_^_5_^_</span>\u3068<span translate=no>_^_6_^_</span>) \u306e\u5f62\u306f\u3067\u3059<span translate=no>_^_7_^_</span>\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>,<span translate=no>_^_1_^_</span>, and <span translate=no>_^_2_^_</span> all should be equal to <span translate=no>_^_3_^_</span> for token mixing </p>\n": "<p><span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u3001<span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u305d\u3057\u3066\u30c8\u30fc\u30af\u30f3\u306e\u30df\u30ad\u30b7\u30f3\u30b0\u3067\u306f\u3059\u3079\u3066\u304c\u7b49\u3057\u304f\u306a\u3051\u308c\u3070\u306a\u308a\u307e\u305b\u3093</p>\n",
|
||||
"<p>Apply the Fourier transform along the hidden (embedding) dimension <span translate=no>_^_0_^_</span></p>\n<p>The output of the Fourier transform is a tensor of <a href=\"https://pytorch.org/docs/stable/complex_numbers.html\">complex numbers</a>. </p>\n": "<p>\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u975e\u8868\u793a (\u57cb\u3081\u8fbc\u307f) \u6b21\u5143\u306b\u6cbf\u3063\u3066\u9069\u7528\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n<p><a href=\"https://pytorch.org/docs/stable/complex_numbers.html\">\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306e\u51fa\u529b\u306f\u8907\u7d20\u6570\u306e\u30c6\u30f3\u30bd\u30eb\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>Apply the Fourier transform along the sequence dimension <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u6b21\u5143\u306b\u6cbf\u3063\u3066\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3092\u9069\u7528\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Assign to <span translate=no>_^_0_^_</span> for clarity </p>\n": "<p><span translate=no>_^_0_^_</span>\u308f\u304b\u308a\u3084\u3059\u3044\u3088\u3046\u306b\u5272\u308a\u5f53\u3066\u308b</p>\n",
|
||||
"<p>Get the real component <span translate=no>_^_0_^_</span> </p>\n": "<p>\u672c\u7269\u306e\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8\u3092\u624b\u306b\u5165\u308c\u3088\u3046 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Token mixing doesn't support masking. i.e. all tokens will see all other token embeddings. </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u30df\u30ad\u30b7\u30f3\u30b0\u306f\u30de\u30b9\u30ad\u30f3\u30b0\u3092\u30b5\u30dd\u30fc\u30c8\u3057\u3066\u3044\u307e\u305b\u3093\u3002\u3064\u307e\u308a\u3001\u3059\u3079\u3066\u306e\u30c8\u30fc\u30af\u30f3\u306b\u4ed6\u306e\u3059\u3079\u3066\u306e\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u304c\u8868\u793a\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408",
|
||||
"This is an annotated implementation/tutorial of FNet in PyTorch.": "\u3053\u308c\u306f PyTorch \u306b\u304a\u3051\u308b FNet \u306e\u30a2\u30ce\u30c6\u30fc\u30b7\u30e7\u30f3\u4ed8\u304d\u306e\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"<h1>FNet: Mixing Tokens with Fourier Transforms</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"../mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a <span translate=no>_^_0_^_</span> more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>.</p>\n<h2>Mixing tokens with two Fourier transforms</h2>\n<p>We apply Fourier transform along the hidden dimension (embedding dimension) and then along the sequence dimension.</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>where <span translate=no>_^_2_^_</span> is the embedding input, <span translate=no>_^_3_^_</span> stands for the fourier transform and <span translate=no>_^_4_^_</span> stands for the real component in complex numbers.</p>\n<p>This is very simple to implement on PyTorch - just 1 line of code. The paper suggests using a precomputed DFT matrix and doing matrix multiplication to get the Fourier transformation.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for using a FNet based model for classifying <a href=\"https://paperswithcode.com/dataset/ag-news\">AG News</a>.</p>\n": "<h1>FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408</h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/2105.03824\">FNet\uff1a\u5c06\u4ee3\u5e01\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408\u300b\u7684 PyTor</a> <a href=\"https://pytorch.org\">ch</a> \u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u7528\u4e24\u4e2a<a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u5085\u91cc\u53f6\u53d8</a>\u6362\u53d6\u4ee3\u4e86<a href=\"../mha.html\">\u81ea\u6211\u6ce8\u610f\u529b\u5c42</a>\uff0c\u4ee5<em>\u6df7\u5408</em>\u4ee4\u724c\u3002\u8fd9\u6bd4\u81ea\u6211\u6ce8\u610f\u529b<span translate=no>_^_0_^_</span>\u66f4\u6709\u6548\u3002\u5728 GLUE <a href=\"https://paperswithcode.com/dataset/glue\">\u57fa\u51c6\u6d4b\u8bd5</a>\u4e2d\uff0c<a href=\"https://paperswithcode.com/method/bert\">BERT</a> \u4f7f\u7528\u5b83\u800c\u4e0d\u662f\u81ea\u6211\u6ce8\u610f\u529b\u7684\u51c6\u786e\u6027\u635f\u5931\u7ea6\u4e3a92\uff05\u3002</p>\n<h2>\u5c06\u4ee4\u724c\u4e0e\u4e24\u4e2a\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408</h2>\n<p>\u6211\u4eec\u6cbf\u9690\u85cf\u7ef4\u5ea6\uff08\u5d4c\u5165\u7ef4\u5ea6\uff09\u5e94\u7528\u5085\u91cc\u53f6\u53d8\u6362\uff0c\u7136\u540e\u6cbf\u5e8f\u5217\u7ef4\u5ea6\u5e94\u7528\u5085\u91cc\u53f6\u53d8\u6362\u3002</p>\n<p><span translate=no>_^_1_^_</span></p>\n<p>\u5176\u4e2d<span translate=no>_^_2_^_</span>\u662f\u5d4c\u5165\u8f93\u5165\uff0c<span translate=no>_^_3_^_</span>\u4ee3\u8868\u5085\u91cc\u53f6\u53d8\u6362\uff0c<span translate=no>_^_4_^_</span>\u4ee3\u8868\u590d\u6570\u4e2d\u7684\u5b9e\u5206\u91cf\u3002</p>\n<p>\u8fd9\u5728 PyTorch \u4e0a\u5b9e\u73b0\u8d77\u6765\u975e\u5e38\u7b80\u5355-\u53ea\u9700\u4e00\u884c\u4ee3\u7801\u3002\u672c\u6587\u5efa\u8bae\u4f7f\u7528\u9884\u5148\u8ba1\u7b97\u7684DFT\u77e9\u9635\u5e76\u8fdb\u884c\u77e9\u9635\u4e58\u6cd5\u6765\u83b7\u5f97\u5085\u91cc\u53f6\u53d8\u6362\u3002</p>\n<p><a href=\"experiment.html\">\u4ee5\u4e0b\u662f\u4f7f\u7528\u57fa\u4e8e FNet \u7684\u6a21\u578b\u5bf9 <a href=\"https://paperswithcode.com/dataset/ag-news\">AG News</a> \u8fdb\u884c\u5206\u7c7b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
|
||||
"<h2>FNet - Mix tokens</h2>\n<p>This module simply implements <span translate=no>_^_0_^_</span></p>\n<p>The structure of this module is made similar to a <a href=\"../mha.html\">standard attention module</a> so that we can simply replace it.</p>\n": "<h2>FNet-\u6df7\u5408\u4ee3\u5e01</h2>\n<p>\u8fd9\u4e2a\u6a21\u5757\u7b80\u5355\u5730\u5b9e\u73b0\u4e86<span translate=no>_^_0_^_</span></p>\n<p>\u8be5\u6a21\u5757\u7684\u7ed3\u6784\u7c7b\u4f3c\u4e8e<a href=\"../mha.html\">\u6807\u51c6\u7684\u6ce8\u610f\u529b\u6a21\u5757</a>\uff0c\u56e0\u6b64\u6211\u4eec\u53ef\u4ee5\u7b80\u5355\u5730\u66ff\u6362\u5b83\u3002</p>\n",
|
||||
"<p> The <a href=\"../mha.html\">normal attention module</a> can be fed with different token embeddings for <span translate=no>_^_0_^_</span>,<span translate=no>_^_1_^_</span>, and <span translate=no>_^_2_^_</span> and a mask.</p>\n<p>We follow the same function signature so that we can replace it directly.</p>\n<p>For FNet mixing, <span translate=no>_^_3_^_</span> and masking is not possible. Shape of <span translate=no>_^_4_^_</span> (and <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>) is <span translate=no>_^_7_^_</span>.</p>\n": "<p><a href=\"../mha.html\">\u666e\u901a\u6ce8\u610f\u529b\u6a21\u5757</a>\u53ef\u4ee5\u4f7f\u7528<span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u548c\u7684\u4e0d\u540c\u4ee4\u724c\u5d4c\u5165<span translate=no>_^_2_^_</span>\u4ee5\u53ca\u63a9\u7801\u8fdb\u884c\u9988\u9001\u3002</p>\n<p>\u6211\u4eec\u9075\u5faa\u76f8\u540c\u7684\u51fd\u6570\u7b7e\u540d\uff0c\u4ee5\u4fbf\u6211\u4eec\u53ef\u4ee5\u76f4\u63a5\u66ff\u6362\u5b83\u3002</p>\n<p>\u5bf9\u4e8e FNet \u6df7\u5408<span translate=no>_^_3_^_</span>\uff0c\u5c4f\u853d\u662f\u4e0d\u53ef\u80fd\u7684\u3002<span translate=no>_^_4_^_</span>\uff08\u548c<span translate=no>_^_5_^_</span>\u548c<span translate=no>_^_6_^_</span>\uff09\u7684\u5f62\u72b6\u4e3a<span translate=no>_^_7_^_</span>\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>,<span translate=no>_^_1_^_</span>, and <span translate=no>_^_2_^_</span> all should be equal to <span translate=no>_^_3_^_</span> for token mixing </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3001\uff0c<span translate=no>_^_3_^_</span>\u5bf9\u4e8e\u4ee4\u724c\u6df7\u5408\uff0cal<span translate=no>_^_2_^_</span> l \u5e94\u8be5\u7b49\u4e8e</p>\n",
|
||||
"<p>Apply the Fourier transform along the hidden (embedding) dimension <span translate=no>_^_0_^_</span></p>\n<p>The output of the Fourier transform is a tensor of <a href=\"https://pytorch.org/docs/stable/complex_numbers.html\">complex numbers</a>. </p>\n": "<p>\u6cbf\u9690\u85cf\uff08\u5d4c\u5165\uff09\u7ef4\u5ea6\u5e94\u7528\u5085\u91cc\u53f6\u53d8\u6362<span translate=no>_^_0_^_</span></p>\n<p>\u5085\u91cc\u53f6\u53d8\u6362\u7684\u8f93\u51fa\u662f<a href=\"https://pytorch.org/docs/stable/complex_numbers.html\">\u590d\u6570</a>\u5f20\u91cf\u3002</p>\n",
|
||||
"<p>Apply the Fourier transform along the sequence dimension <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6cbf\u5e8f\u5217\u7ef4\u5ea6\u5e94\u7528\u5085\u91cc\u53f6\u53d8\u6362<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Assign to <span translate=no>_^_0_^_</span> for clarity </p>\n": "<p>\u4e3a\u4e86\u6e05\u695a\u8d77\u89c1\uff0c<span translate=no>_^_0_^_</span>\u8bf7\u5206\u914d\u7ed9</p>\n",
|
||||
"<p>Get the real component <span translate=no>_^_0_^_</span> </p>\n": "<p>\u83b7\u53d6\u771f\u6b63\u7684\u7ec4\u4ef6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Token mixing doesn't support masking. i.e. all tokens will see all other token embeddings. </p>\n": "<p>\u4ee4\u724c\u6df7\u5408\u4e0d\u652f\u6301\u63a9\u7801\u3002\u5373\u6240\u6709\u4ee4\u724c\u90fd\u5c06\u770b\u5230\u6240\u6709\u5176\u4ed6\u4ee4\u724c\u5d4c\u5165\u3002</p>\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408",
|
||||
"This is an annotated implementation/tutorial of FNet in PyTorch.": "\u8fd9\u662f PyTorch \u4e2d FNet \u7684\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0/\u6559\u7a0b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">FNet</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">FNet model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_classification.html\">general training loop and configurations for AG News classification task</a>.</p>\n": "<h1><a href=\"index.html\">FNet \u5b9f\u9a13</a></h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f\u3001FNet\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u306ePyTorch\u5b9f\u9a13\u3067\u3059\u3002</a></p>\n<p>\u3053\u308c\u306f\u3001<a href=\"../../experiments/nlp_classification.html\">AG News\u5206\u985e\u30bf\u30b9\u30af\u306e\u4e00\u822c\u7684\u306a\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u69cb\u6210\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h1>Transformer based classifier model</h1>\n": "<h1>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30d9\u30fc\u30b9\u306e\u5206\u985e\u5668\u30e2\u30c7\u30eb</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u4ee5\u4e0b\u304b\u3089\u7d99\u627f\u3055\u308c\u307e\u3059 <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create <span translate=no>_^_0_^_</span> module that can replace the self-attention in <a href=\"../models.html#TransformerLayer\">transformer encoder layer</a> .</p>\n": "<p><span translate=no>_^_0_^_</span><a href=\"../models.html#TransformerLayer\">\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u5c64\u306e\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u306b\u4ee3\u308f\u308b\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u4f5c\u6210\u3057\u3066\u304f\u3060\u3055\u3044</a>\u3002</p>\n",
|
||||
"<p> Create classification model</p>\n": "<p>\u5206\u985e\u30e2\u30c7\u30eb\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Classification model </p>\n": "<p>\u5206\u985e\u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Get logits for classification.</p>\n<p>We set the <span translate=no>_^_0_^_</span> token at the last position of the sequence. This is extracted by <span translate=no>_^_1_^_</span>, where <span translate=no>_^_2_^_</span> is of shape <span translate=no>_^_3_^_</span> </p>\n": "<p>\u5206\u985e\u7528\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97\u3057\u307e\u3059\u3002</p>\n<p><span translate=no>_^_0_^_</span>\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u6700\u5f8c\u306e\u4f4d\u7f6e\u306b\u30c8\u30fc\u30af\u30f3\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u5f62\u72b6\u304c\u3069\u3053\u306b\u3042\u308b\u304b\u306b\u3088\u3063\u3066\u62bd\u51fa\u3055\u308c\u307e\u3059 <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3057\u307e\u3059\uff08\u30c8\u30ec\u30fc\u30ca\u30fc\u306fRNN\u3067\u3082\u4f7f\u7528\u3055\u308c\u308b\u305f\u3081\u30012\u756a\u76ee\u306e\u5024\u306f\u72b6\u614b\u7528\u3067\u3059\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u5909\u5727\u5668\u69cb\u6210 (\u30c7\u30d5\u30a9\u30eb\u30c8\u3068\u540c\u3058)</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use <a href=\"index.html\">FNet</a> instead of self-a ttention </p>\n": "<p><a href=\"index.html\">\u81ea\u5df1\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u306e\u4ee3\u308f\u308a\u306bFNet\u3092\u4f7f\u3046</a></p>\n",
|
||||
"<p>Use world level tokenizer </p>\n": "<p>\u30ef\u30fc\u30eb\u30c9\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../models.html#Encoder\">\u5909\u5727\u5668\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span><a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u306f\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3059 (\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u4ed8\u304d)</a></li>\n</ul><li><span translate=no>_^_2_^_</span><a href=\"../models.html#Generator\">\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u6700\u5f8c\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u3067\u3059</a>\u3002</li>\n",
|
||||
"FNet Experiment": "FNet \u5b9f\u9a13",
|
||||
"This experiment trains a FNet based model on AG News dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001AG News\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u57fa\u3065\u3044\u3066FNet\u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">FNet</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">FNet model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_classification.html\">general training loop and configurations for AG News classification task</a>.</p>\n": "<h1><a href=\"index.html\">FNet</a> \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">FNet \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf PyTorch \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p>\u0db8\u0dd9\u0dba\u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda <a href=\"../../experiments/nlp_classification.html\">AG \u0db4\u0dca\u0dbb\u0dc0\u0dd8\u0dad\u0dca\u0dad\u0dd2 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0db8\u0dad \u0dba. </p>\n",
|
||||
"<h1>Transformer based classifier model</h1>\n": "<h1>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0db1\u0dca\u0db1\u0dda <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create <span translate=no>_^_0_^_</span> module that can replace the self-attention in <a href=\"../models.html#TransformerLayer\">transformer encoder layer</a> .</p>\n": "<p> <a href=\"../models.html#TransformerLayer\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda</a> \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 <span translate=no>_^_0_^_</span> \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p> Create classification model</p>\n": "<p> \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p>Classification model </p>\n": "<p>\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get logits for classification.</p>\n<p>We set the <span translate=no>_^_0_^_</span> token at the last position of the sequence. This is extracted by <span translate=no>_^_1_^_</span>, where <span translate=no>_^_2_^_</span> is of shape <span translate=no>_^_3_^_</span> </p>\n": "<p>\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1. </p>\n<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dba\u0dda \u0d85\u0db4\u0dd2 <span translate=no>_^_0_^_</span> \u0da7\u0ddd\u0d9a\u0db1\u0dba \u0dc3\u0d9a\u0dc3\u0dca \u0d9a\u0dbb\u0db8\u0dd4. \u0db8\u0dd9\u0dba \u0d8b\u0db4\u0dd4\u0da7\u0dcf \u0d9c\u0db1\u0dd4 \u0dbd\u0db6\u0db1\u0dca\u0db1\u0dda <span translate=no>_^_1_^_</span>\u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca <span translate=no>_^_2_^_</span> \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf </p>? <span translate=no>_^_3_^_</span>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dda\u0dad\u0db1 \u0d9a\u0dca\u0dbb\u0db8 \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0dcf\u0db7\u0db4\u0dca\u0dbb\u0dad\u0dd2 results \u0dbd (\u0daf\u0dd9\u0dc0\u0db1 \u0d85\u0d9c\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dda, \u0db8\u0db1\u0dca\u0daf \u0d85\u0db4\u0d9c\u0dda \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4 RNs \u0dc3\u0db8\u0d9f \u0daf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2) </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca (\u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dbd\u0dd9\u0dc3) </p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use <a href=\"index.html\">FNet</a> instead of self-a ttention </p>\n": "<p>\u0dc3\u0dca\u0dc0\u0dba\u0d82-\u0d87\u0dbd\u0dc0\u0dd3\u0db8\u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7 <a href=\"index.html\">FNet</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use world level tokenizer </p>\n": "<p>\u0dbd\u0ddd\u0d9a\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"../models.html#Encoder\">\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a> </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0da7\u0ddd\u0d9a\u0db1\u0dca <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba (\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab \u0dc3\u0db8\u0d9f)</a> </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1 <a href=\"../models.html#Generator\">\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dba\u0dd2</a> . </li></ul>\n",
|
||||
"FNet Experiment": "FNet \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains a FNet based model on AG News dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 AG News \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db8\u0dad FNet \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">FNet</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">FNet model</a>.</p>\n<p>This is based on <a href=\"../../experiments/nlp_classification.html\">general training loop and configurations for AG News classification task</a>.</p>\n": "<h1><a href=\"index.html\">FNet</a> \u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u4e2a\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3 <a href=\"index.html\">FNet \u6a21\u578b</a>\u3002</p>\n<p>\u8fd9\u662f\u57fa\u4e8e <a href=\"../../experiments/nlp_classification.html\">AG News \u5206\u7c7b\u4efb\u52a1\u7684\u4e00\u822c\u8bad\u7ec3\u5faa\u73af\u548c\u914d\u7f6e</a>\u3002</p>\n",
|
||||
"<h1>Transformer based classifier model</h1>\n": "<h1>\u57fa\u4e8e\u53d8\u538b\u5668\u7684\u5206\u7c7b\u5668\u6a21\u578b</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea <a href=\"../../experiments/nlp_classification.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create <span translate=no>_^_0_^_</span> module that can replace the self-attention in <a href=\"../models.html#TransformerLayer\">transformer encoder layer</a> .</p>\n": "<p>\u521b\u5efa\u53ef\u4ee5\u53d6\u4ee3<a href=\"../models.html#TransformerLayer\">\u53d8\u538b\u5668\u7f16\u7801\u5668\u5c42</a>\u4e2d\u7684\u81ea\u6211\u6ce8\u610f\u529b\u7684<span translate=no>_^_0_^_</span>\u6a21\u5757\u3002</p>\n",
|
||||
"<p> Create classification model</p>\n": "<p>\u521b\u5efa\u5206\u7c7b\u6a21\u578b</p>\n",
|
||||
"<p>Classification model </p>\n": "<p>\u5206\u7c7b\u6a21\u578b</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Get logits for classification.</p>\n<p>We set the <span translate=no>_^_0_^_</span> token at the last position of the sequence. This is extracted by <span translate=no>_^_1_^_</span>, where <span translate=no>_^_2_^_</span> is of shape <span translate=no>_^_3_^_</span> </p>\n": "<p>\u83b7\u53d6\u5206\u7c7b\u65e5\u5fd7\u3002</p>\n<p>\u6211\u4eec\u5c06\u4ee4<span translate=no>_^_0_^_</span>\u724c\u8bbe\u7f6e\u5728\u5e8f\u5217\u7684\u6700\u540e\u4e00\u4e2a\u4f4d\u7f6e\u3002\u8fd9\u662f\u7531\uff0cwher<span translate=no>_^_1_^_</span> e<span translate=no>_^_2_^_</span> \u662f\u5f62\u72b6\u63d0\u53d6</p>\u7684<span translate=no>_^_3_^_</span>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f7f\u7528\u4f4d\u7f6e\u7f16\u7801\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c\uff08\u7b2c\u4e8c\u4e2a\u503c\u7528\u4e8e\u72b6\u6001\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u8bad\u7ec3\u5668\u4e5f\u4e0e RNN \u4e00\u8d77\u4f7f\u7528\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u53d8\u538b\u5668\u914d\u7f6e\uff08\u4e0e\u9ed8\u8ba4\u503c\u76f8\u540c\uff09</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use <a href=\"index.html\">FNet</a> instead of self-a ttention </p>\n": "<p>\u4f7f\u7528 <a href=\"index.html\">FNet</a> \u800c\u4e0d\u662f\u81ea\u6211\u5173\u6ce8</p>\n",
|
||||
"<p>Use world level tokenizer </p>\n": "<p>\u4f7f\u7528\u4e16\u754c\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668<a href=\"../models.html#Encoder\">\u7f16\u7801\u5668</a></li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c<a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u5d4c\u5165\u6a21\u5757\uff08\u5e26\u6709\u4f4d\u7f6e\u7f16\u7801\uff09</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u7ed9<a href=\"../models.html#Generator\">\u51fa logit \u7684\u6700\u540e\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42</a>\u3002</li></ul>\n",
|
||||
"FNet Experiment": "FNet \u5b9e\u9a8c",
|
||||
"This experiment trains a FNet based model on AG News dataset.": "\u8be5\u5b9e\u9a8c\u57fa\u4e8eAG News\u6570\u636e\u96c6\u8bad\u7ec3\u4e00\u4e2a\u57fa\u4e8eFNet\u7684\u6a21\u578b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers//mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408</a></h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.03824\">FNet: \u30c8\u30fc\u30af\u30f3\u3092\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u3068\u6df7\u5408\u3059\u308b\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n<p><em>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<a href=\"https://nn.labml.ai/transformers//mha.html\"><a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u81ea\u5df1\u6ce8\u610f\u5c64\u30922\u3064\u306e\u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u7f6e\u304d\u63db\u3048\u3066\u30c8\u30fc\u30af\u30f3\u3092\u6df7\u5408\u3057\u307e\u3059</a></a>\u3002</em>\u3053\u308c\u306f\u81ea\u5df1\u51e6\u7406\u3088\u308a\u30827\u500d\u52b9\u7387\u7684\u3067\u3059\u3002<a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE</a> \u30d9\u30f3\u30c1\u30de\u30fc\u30af\u3067\u306f\u3001\u81ea\u5df1\u6ce8\u610f\u3088\u308a\u3082\u3053\u308c\u3092\u4f7f\u7528\u3057\u305f\u5834\u5408\u306e\u7cbe\u5ea6\u306e\u4f4e\u4e0b\u306f\u7d04 92%</p> \u3067\u3059\u3002\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet: \u30d5\u30fc\u30ea\u30a8\u5909\u63db\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u6df7\u5408"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers//mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: \u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0db4\u0dbb\u0dd2\u0dab\u0dcf\u0db8\u0db1\u0dba \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1 \u0db8\u0dd2\u0dc1\u0dca\u0dbb \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a></h1>\n<p>\u0db8\u0dd9\u0dba\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://arxiv.org/abs/2105.03824\">FNet \u0dc4\u0dd2 <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8: \u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dca \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1 \u0db8\u0dd2\u0dc1\u0dca\u0dbb</a> \u0d9a\u0dd2\u0dbb\u0dd3\u0db8. </p>\n<p>\u0db8\u0dd9\u0db8\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://nn.labml.ai/transformers//mha.html\">\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dad\u0dbb\u0dba</a> \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2 <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dd9\u0d9a\u0d9a\u0dca \u0da7\u0ddd\u0d9a\u0db1 <em>\u0db8\u0dd2\u0dc1\u0dca\u0dbb</em> \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba</a> \u0d9a\u0dbb\u0dba\u0dd2. \u0db8\u0dd9\u0dba 7X \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0da7 \u0dc0\u0da9\u0dcf \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8 \u0dc0\u0dda. \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dd9\u0db1\u0dca \u0db8\u0dd9\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db1\u0dd0\u0dad\u0dd2\u0dc0\u0dd3\u0db8 GLUE <a href=\"https://paperswithcode.com/dataset/glue\">\u0db8\u0dd2\u0dab\u0dd4\u0db8\u0dca \u0daf\u0dab\u0dca\u0da9\u0dda</a> <a href=\"https://paperswithcode.com/method/bert\">BERT</a> \u0dc3\u0db3\u0dc4\u0dcf 92% \u0d9a\u0dca \u0db4\u0db8\u0dab \u0dc0\u0dda. </p>\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet: \u0dc6\u0dd6\u0dbb\u0dd2\u0dba\u0dbb\u0dca \u0db4\u0dbb\u0dd2\u0dab\u0dcf\u0db8\u0db1\u0dba \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1 \u0db8\u0dd2\u0dc1\u0dca\u0dbb \u0d9a\u0dd2\u0dbb\u0dd3\u0db8"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers//mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408</a></h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/2105.03824\">FNet\uff1a\u5c06\u4ee3\u5e01\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408\u300b\u7684 PyTor</a> <a href=\"https://pytorch.org\">ch</a> \u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u7528\u4e24\u4e2a<a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u5085\u91cc\u53f6\u53d8</a>\u6362\u53d6\u4ee3\u4e86<a href=\"https://nn.labml.ai/transformers//mha.html\">\u81ea\u6211\u6ce8\u610f\u529b\u5c42</a>\uff0c\u4ee5<em>\u6df7\u5408</em>\u4ee4\u724c\u3002\u8fd9\u6bd4\u81ea\u6211\u6ce8\u610f\u529b\u9ad87\u500d\u3002\u5728 GLUE <a href=\"https://paperswithcode.com/dataset/glue\">\u57fa\u51c6\u6d4b\u8bd5</a>\u4e2d\uff0c<a href=\"https://paperswithcode.com/method/bert\">BERT</a> \u4f7f\u7528\u5b83\u800c\u4e0d\u662f\u81ea\u6211\u6ce8\u610f\u529b\u7684\u51c6\u786e\u6027\u635f\u5931\u7ea6\u4e3a92\uff05\u3002</p>\n",
|
||||
"FNet: Mixing Tokens with Fourier Transforms": "FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408"
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<ul><li><a href=\"experiment.html\">Experiment that uses <span translate=no>_^_0_^_</span></a> </li>\n<li><a href=\"simple.html\">Simpler version from scratch</a></li></ul>\n": "<h1>\u30b2\u30fc\u30c6\u30c3\u30c9\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8\u3068\u30d0\u30ea\u30a2\u30f3\u30c8</h1>\n<ul><li><a href=\"experiment.html\">\u3092\u4f7f\u7528\u3059\u308b\u5b9f\u9a13 <span translate=no>_^_0_^_</span></a></li>\n<li><a href=\"simple.html\">\u30bc\u30ed\u304b\u3089\u306e\u30b7\u30f3\u30d7\u30eb\u306a\u30d0\u30fc\u30b8\u30e7\u30f3</a></li></ul>\n",
|
||||
"Gated Linear Units and Variants": "\u30b2\u30fc\u30c6\u30c3\u30c9\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8\u3068\u30d0\u30ea\u30a2\u30f3\u30c8",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u4f4d\u7f6e\u5358\u4f4d\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af (FFN) \u306e\u30b2\u30fc\u30c8\u7dda\u5f62\u5358\u4f4d\u3068\u30d0\u30ea\u30a2\u30f3\u30c8\u3092\u4f7f\u7528\u3057\u3066\u81ea\u5df1\u56de\u5e30\u5909\u5727\u5668\u306b\u5b66\u7fd2\u3055\u305b\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<ul><li><a href=\"glu_variants/experiment.html\">Experiment that uses <span translate=no>_^_0_^_</span></a> </li>\n<li><a href=\"glu_variants/simple.html\">Simpler version from scratch</a></li></ul>\n": "<h1>\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf</h1>\n<ul><li><a href=\"glu_variants/experiment.html\">\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 <span translate=no>_^_0_^_</span></a> </li>\n<li><a href=\"glu_variants/simple.html\">\u0db8\u0dd4\u0dbd \u0dc3\u0dd2\u0da7 \u0dc3\u0dbb\u0dbd \u0d85\u0db1\u0dd4\u0dc0\u0dcf\u0daf\u0dba</a></li></ul>\n",
|
||||
"Gated Linear Units and Variants": "\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0ddd\u0dc2\u0d9a \u0da2\u0dcf\u0dbd\u0dba (FFN) \u0dc3\u0db3\u0dc4\u0dcf \u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf \u0dc3\u0db8\u0d9f \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1."
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<ul><li><a href=\"experiment.html\">Experiment that uses <span translate=no>_^_0_^_</span></a> </li>\n<li><a href=\"simple.html\">Simpler version from scratch</a></li></ul>\n": "<h1>\u95e8\u63a7\u7ebf\u6027\u5355\u5143\u548c\u53d8\u4f53</h1>\n<ul><li><a href=\"experiment.html\">\u4f7f\u7528\u7684\u5b9e\u9a8c<span translate=no>_^_0_^_</span></a></li>\n<li><a href=\"simple.html\">\u4ece\u5934\u5f00\u59cb\u66f4\u7b80\u5355\u7684\u7248\u672c</a></li></ul>\n",
|
||||
"Gated Linear Units and Variants": "\u95e8\u63a7\u7ebf\u6027\u5355\u4f4d\u548c\u53d8\u4f53",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u4f7f\u7528\u95e8\u63a7\u7ebf\u6027\u5355\u5143\u548c\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc (FFN) \u53d8\u4f53\u8bad\u7ec3\u81ea\u56de\u5f52\u53d8\u538b\u5668\u3002"
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<p>This trains a simple <a href=\"../../\">transformer</a> model for auto-regression. We try different variants for the <a href=\"../feed_forward\">position-wise feedforward network</a>. The reusable & configurable are defined in <a href=\"configs.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h1>\u30b2\u30fc\u30c6\u30c3\u30c9\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8\u3068\u30d0\u30ea\u30a2\u30f3\u30c8</h1>\n</a><p>\u3053\u308c\u306b\u3088\u308a\u3001<a href=\"../../\">\u81ea\u52d5\u56de\u5e30\u7528\u306e\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u304c\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3055\u308c\u307e\u3059\u3002<a href=\"../feed_forward\">\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306b\u306f\u3055\u307e\u3056\u307e\u306a\u30d0\u30ea\u30a8\u30fc\u30b7\u30e7\u30f3\u3092\u8a66\u3057\u307e\u3059</a>\u3002\u518d\u5229\u7528\u53ef\u80fd\u304a\u3088\u3073\u8a2d\u5b9a\u53ef\u80fd\u306a\u3082\u306e\u306f\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u307e\u3059</p>\u3002<a href=\"configs.html\"><span translate=no>_^_0_^_</span></a>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52d5\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u8a2d\u5b9a\u306f\u3001\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u305f\u3068\u304d\u306b\u4e0a\u66f8\u304d\u3067\u304d\u3001\u307e\u305f\u4e0a\u66f8\u304d\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p> Initialize the <a href=\"../configs.html\">configurable transformer</a> encoder for our autoregressive model.</p>\n": "<p><a href=\"../configs.html\">\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u306e\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u3092\u521d\u671f\u5316\u3057\u307e\u3059</a>\u3002</p>\n",
|
||||
"<p> Initialize the auto-regressive model</p>\n": "<p>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u3092\u521d\u671f\u5316</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u3059\u308b\u8a2d\u5b9a\u306e\u8f9e\u66f8</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create subsequent mask, so that the transformer can only pay attention to past tokens. </p>\n": "<p>\u6b21\u306e\u30de\u30b9\u30af\u3092\u4f5c\u6210\u3057\u3066\u3001\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u304c\u904e\u53bb\u306e\u30c8\u30fc\u30af\u30f3\u306b\u3057\u304b\u6ce8\u76ee\u3067\u304d\u306a\u3044\u3088\u3046\u306b\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Embed the tokens (<span translate=no>_^_0_^_</span>) and run it through the the transformer </p>\n": "<p>\u30c8\u30fc\u30af\u30f3 (<span translate=no>_^_0_^_</span>) \u3092\u57cb\u3081\u8fbc\u307f\u3001\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306b\u901a\u3057\u307e\u3059</p>\n",
|
||||
"<p>GLU Variant, one of GLU, Bilinear, ReGLU, GEGLU, SwiGLU</p>\n<p>These are defined in the <a href=\"../configs.html#FFN\">configurable FFN</a> implementation </p>\n": "<p>GLU \u30d0\u30ea\u30a2\u30f3\u30c8\u3001GLU\u3001\u30d0\u30a4\u30ea\u30cb\u30a2\u3001RegLU\u3001GEGLU\u3001SwiGLU \u306e\u3044\u305a\u308c\u304b</p>\n<p><a href=\"../configs.html#FFN\">\u3053\u308c\u3089\u306f\u8a2d\u5b9a\u53ef\u80fd\u306a FFN \u5b9f\u88c5\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Next token generation layer; this give logits of the the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u751f\u6210\u30ec\u30a4\u30e4\u30fc\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u304c\u8fd4\u3055\u308c\u307e\u3059</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>This is needed to initialize models </p>\n": "<p>\u3053\u308c\u306f\u30e2\u30c7\u30eb\u3092\u521d\u671f\u5316\u3059\u308b\u305f\u3081\u306b\u5fc5\u8981\u3067\u3059</p>\n",
|
||||
"<p>This will be initialized on the first call </p>\n": "<p>\u3053\u308c\u306f\u6700\u521d\u306e\u547c\u3073\u51fa\u3057\u3067\u521d\u671f\u5316\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"<p>Transformer based encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30d9\u30fc\u30b9\u306e\u30a8\u30f3\u30b3\u30fc\u30c0</p>\n",
|
||||
"<p>Transformer configurations </p>\n": "<p>\u5909\u5727\u5668\u69cb\u6210</p>\n",
|
||||
"Gated Linear Units and Variants": "\u30b2\u30fc\u30c6\u30c3\u30c9\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8\u3068\u30d0\u30ea\u30a2\u30f3\u30c8",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u4f4d\u7f6e\u5358\u4f4d\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af (FFN) \u306e\u30b2\u30fc\u30c8\u7dda\u5f62\u5358\u4f4d\u3068\u30d0\u30ea\u30a2\u30f3\u30c8\u3092\u4f7f\u7528\u3057\u3066\u81ea\u5df1\u56de\u5e30\u5909\u5727\u5668\u306b\u5b66\u7fd2\u3055\u305b\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<p>This trains a simple <a href=\"../../\">transformer</a> model for auto-regression. We try different variants for the <a href=\"../feed_forward\">position-wise feedforward network</a>. The reusable & configurable are defined in <a href=\"configs.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h1>\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf</h1>\n</a> <p>\u0db8\u0dd9\u0dba\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dbb\u0dbd <a href=\"../../\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2. <a href=\"../feed_forward\">\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0ddd\u0dc2\u0d9a \u0da2\u0dcf\u0dbd\u0dba</a>\u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 \u0dc0\u0dd2\u0dc0\u0dd2\u0db0 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf \u0d8b\u0dad\u0dca\u0dc3\u0dcf\u0dc4 \u0d9a\u0dbb\u0db8\u0dd4. \u0db1\u0dd0\u0dc0\u0dad \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad <a href=\"configs.html\"><span translate=no>_^_0_^_</span></a>. </p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0d85\u0db4\u0dd2\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1 \u0dc0\u0dd2\u0da7 \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3 \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0d85\u0dad\u0dbb \u0d91\u0dba \u0d85\u0db0\u0dd2\u0d9a \u0dbd\u0dd9\u0dc3 \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0dc0\u0db1\u0dd4 \u0d87\u0dad</p>\n",
|
||||
"<p> Initialize the <a href=\"../configs.html\">configurable transformer</a> encoder for our autoregressive model.</p>\n": "<p> \u0d85\u0db4\u0d9c\u0dda\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dc3\u0db3\u0dc4\u0dcf <a href=\"../configs.html\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca</a> \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p> Initialize the auto-regressive model</p>\n": "<p> \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf\u0dba\u0dcf\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc1\u0db6\u0dca\u0daf\u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dca </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create subsequent mask, so that the transformer can only pay attention to past tokens. </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d85\u0dad\u0dd3\u0dad \u0da7\u0ddd\u0d9a\u0db1 \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0db4\u0db8\u0dab\u0d9a\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0dc0\u0db1 \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Embed the tokens (<span translate=no>_^_0_^_</span>) and run it through the the transformer </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 (<span translate=no>_^_0_^_</span>) \u0dc3\u0dc4 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0d91\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>GLU Variant, one of GLU, Bilinear, ReGLU, GEGLU, SwiGLU</p>\n<p>These are defined in the <a href=\"../configs.html#FFN\">configurable FFN</a> implementation </p>\n": "<p>GLU\u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf\u0dba, GLU, \u0daf\u0dca\u0dc0\u0dd2\u0dbd\u0dd3\u0db1\u0dd2\u0dba\u0dbb\u0dca, \u0dbb\u0dd9\u0d9c\u0dca\u0dbd\u0dd6, GEGLU, \u0dc3\u0dca\u0dc0\u0dd2\u0d9c\u0dca\u0dbd\u0dd6 \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0d91\u0d9a\u0d9a\u0dca</p>\n<p><a href=\"../configs.html#FFN\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 FFN</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3 \u0db8\u0dda\u0dc0\u0dcf \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad </p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Next token generation layer; this give logits of the the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1 \u0dc3\u0dca\u0dad\u0dbb\u0dba; \u0db8\u0dd9\u0dba \u0d8a\u0dc5\u0d9f \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0dba\u0dd2 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>This is needed to initialize models </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dd9\u0dba \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dda </p>\n",
|
||||
"<p>This will be initialized on the first call </p>\n": "<p>\u0db8\u0dd9\u0dba\u0db4\u0dc5\u0db8\u0dd4 \u0d87\u0db8\u0dad\u0dd4\u0db8\u0dd9\u0db1\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd9\u0dbb\u0dda </p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"<p>Transformer based encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Transformer configurations </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca </p>\n",
|
||||
"Gated Linear Units and Variants": "\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0ddd\u0dc2\u0d9a \u0da2\u0dcf\u0dbd\u0dba (FFN) \u0dc3\u0db3\u0dc4\u0dcf \u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf \u0dc3\u0db8\u0d9f \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1."
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<p>This trains a simple <a href=\"../../\">transformer</a> model for auto-regression. We try different variants for the <a href=\"../feed_forward\">position-wise feedforward network</a>. The reusable & configurable are defined in <a href=\"configs.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h1>\u95e8\u63a7\u7ebf\u6027\u5355\u4f4d\u548c\u53d8\u4f53</h1>\n</a><p>\u8fd9\u5c06\u4e3a\u81ea\u52a8\u56de\u5f52\u8bad\u7ec3\u4e00\u4e2a\u7b80\u5355\u7684<a href=\"../../\">\u53d8\u538b\u5668\u6a21\u578b\u3002\u6211\u4eec\u4e3a<a href=\"../feed_forward\">\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc</a>\u5c1d\u8bd5\u4e0d\u540c\u7684\u53d8\u4f53\u3002\u53ef\u91cd\u7528\u548c\u53ef\u914d\u7f6e\u7684\u5b9a\u4e49\u5728\u4e2d<a href=\"configs.html\"><span translate=no>_^_0_^_</span></a>\u3002</p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52a8\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configs can and will be over-ridden when we start the experiment</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u5f53\u6211\u4eec\u5f00\u59cb\u5b9e\u9a8c\u65f6\uff0c\u9ed8\u8ba4\u914d\u7f6e\u53ef\u4ee5\u800c\u4e14\u5c06\u4f1a\u88ab\u8986\u76d6</p>\n",
|
||||
"<p> Initialize the <a href=\"../configs.html\">configurable transformer</a> encoder for our autoregressive model.</p>\n": "<p>\u4e3a\u6211\u4eec\u7684\u81ea\u56de\u5f52\u6a21\u578b\u521d\u59cb\u5316<a href=\"../configs.html\">\u53ef\u914d\u7f6e\u7684\u53d8\u538b</a>\u5668\u7f16\u7801\u5668\u3002</p>\n",
|
||||
"<p> Initialize the auto-regressive model</p>\n": "<p>\u521d\u59cb\u5316\u81ea\u56de\u5f52\u6a21\u578b</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u8981\u8986\u76d6\u7684\u914d\u7f6e\u5b57\u5178</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Create subsequent mask, so that the transformer can only pay attention to past tokens. </p>\n": "<p>\u521b\u5efa\u540e\u7eed\u63a9\u7801\uff0c\u4ee5\u4fbf\u53d8\u538b\u5668\u53ea\u80fd\u5173\u6ce8\u8fc7\u53bb\u7684\u4ee4\u724c\u3002</p>\n",
|
||||
"<p>Embed the tokens (<span translate=no>_^_0_^_</span>) and run it through the the transformer </p>\n": "<p>\u5d4c\u5165\u4ee4\u724c (<span translate=no>_^_0_^_</span>) \u5e76\u901a\u8fc7\u53d8\u538b\u5668\u8fd0\u884c\u5b83</p>\n",
|
||||
"<p>GLU Variant, one of GLU, Bilinear, ReGLU, GEGLU, SwiGLU</p>\n<p>These are defined in the <a href=\"../configs.html#FFN\">configurable FFN</a> implementation </p>\n": "<p>GLU Variant\uff0cGLU\u3001Bilinear\u3001regLU\u3001GEGLU\u3001SwiGLU \u4e4b\u4e00</p>\n<p>\u8fd9\u4e9b\u662f\u5728<a href=\"../configs.html#FFN\">\u53ef\u914d\u7f6e\u7684 FFN</a> \u5b9e\u73b0\u4e2d\u5b9a\u4e49\u7684</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u751f\u6210\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Next token generation layer; this give logits of the the next token </p>\n": "<p>\u4e0b\u4e00\u4ee3\u5e01\u751f\u6210\u5c42\uff1b\u8fd9\u4f1a\u7ed9\u51fa\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>This is needed to initialize models </p>\n": "<p>\u8fd9\u662f\u521d\u59cb\u5316\u6a21\u578b\u6240\u5fc5\u9700\u7684</p>\n",
|
||||
"<p>This will be initialized on the first call </p>\n": "<p>\u8fd9\u5c06\u5728\u7b2c\u4e00\u6b21\u8c03\u7528\u65f6\u521d\u59cb\u5316</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u6a21\u5757</p>\n",
|
||||
"<p>Transformer based encoder </p>\n": "<p>\u57fa\u4e8e\u53d8\u538b\u5668\u7684\u7f16\u7801\u5668</p>\n",
|
||||
"<p>Transformer configurations </p>\n": "<p>\u53d8\u538b\u5668\u914d\u7f6e</p>\n",
|
||||
"Gated Linear Units and Variants": "\u95e8\u63a7\u7ebf\u6027\u5355\u4f4d\u548c\u53d8\u4f53",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u4f7f\u7528\u95e8\u63a7\u7ebf\u6027\u5355\u5143\u548c\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc (FFN) \u53d8\u4f53\u8bad\u7ec3\u81ea\u56de\u5f52\u53d8\u538b\u5668\u3002"
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<p>This trains a simple <a href=\"../../\">transformer</a> model for auto-regression. We try different variants for the <a href=\"../feed_forward\">position-wise feedforward network</a>.</p>\n<p><em>This is a simpler implementation that doesn't use <a href=\"experiment.html\"><span translate=no>_^_0_^_</span></a> module. We decided to write a simpler implementation to make it easier for readers who are not familiar.</em></p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/glu_variants/simple.ipynb\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u30b2\u30fc\u30c6\u30c3\u30c9\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8\u3068\u30d0\u30ea\u30a2\u30f3\u30c8</h1>\n</a><p>\u3053\u308c\u306b\u3088\u308a\u3001<a href=\"../../\">\u81ea\u52d5\u56de\u5e30\u7528\u306e\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u304c\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3055\u308c\u307e\u3059\u3002<a href=\"../feed_forward\">\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306b\u306f\u3055\u307e\u3056\u307e\u306a\u30d0\u30ea\u30a8\u30fc\u30b7\u30e7\u30f3\u3092\u8a66\u3057\u307e\u3059</a></p>\u3002\n<p><em><a href=\"experiment.html\"><span translate=no>_^_0_^_</span></a>\u3053\u308c\u306f\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u4f7f\u308f\u306a\u3044\u3088\u308a\u5358\u7d14\u306a\u5b9f\u88c5\u3067\u3059\u3002\u6163\u308c\u3066\u3044\u306a\u3044\u8aad\u8005\u306b\u3082\u308f\u304b\u308a\u3084\u3059\u3044\u3088\u3046\u306b\u3001\u3088\u308a\u30b7\u30f3\u30d7\u30eb\u306a\u5b9f\u88c5\u3092\u4f5c\u6210\u3059\u308b\u3053\u3068\u306b\u3057\u307e\u3057\u305f</em></p>\u3002\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/glu_variants/simple.ipynb\"><span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52d5\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Trainer</h2>\n": "<h2>\u30c8\u30ec\u30fc\u30ca\u30fc</h2>\n",
|
||||
"<h3>Configurations</h3>\n": "<h3>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\u5b9a\u671f\u7684\u306b\u30b5\u30f3\u30d7\u30eb\u3092\u751f\u6210\u3059\u308b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u6a5f\u80fd</h3>\n",
|
||||
"<h3>Tiny Shakespeare Dataset</h3>\n": "<h3>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8</h3>\n",
|
||||
"<h3>Train the model</h3>\n": "<h3>\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</h3>\n",
|
||||
"<p> Number of samples in the dataset.</p>\n<p><em>This will read the dataset <span translate=no>_^_0_^_</span> times in a single epoch.</em></p>\n": "<p>\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u30b5\u30f3\u30d7\u30eb\u6570\u3002</p>\n<p><em>\u3053\u308c\u306b\u3088\u308a\u30011 <span translate=no>_^_0_^_</span> \u3064\u306e\u30a8\u30dd\u30c3\u30af\u3067\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u56de\u6570\u304c\u8aad\u307f\u53d6\u3089\u308c\u307e\u3059\u3002</em></p>\n",
|
||||
"<p> Return a sample</p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u3092\u8fd4\u3059</p>\n",
|
||||
"<p> Transform the text into a tensor of ids</p>\n": "<p>\u30c6\u30ad\u30b9\u30c8\u3092 id \u306e\u30c6\u30f3\u30bd\u30eb\u306b\u5909\u63db\u3057\u307e\u3059</p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u30ed\u30ae\u30f3\u30b0\u7528\u306e\u4e88\u6e2c\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u4e88\u6e2c\u3092\u30d7\u30ed\u30f3\u30d7\u30c8\u306b\u8ffd\u52a0</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u52fe\u914d\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate loss </p>\n": "<p>\u640d\u5931\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Character to id (integer) map </p>\n": "<p>\u6587\u5b57\u3092 ID (\u6574\u6570) \u306b\u30de\u30c3\u30d4\u30f3\u30b0</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30af\u30ea\u30a2</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u30af\u30ea\u30c3\u30d7\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u5370\u5237\u7528\u306e\u51fa\u529b\u3092\u53ce\u96c6</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create subsequent mask, so that the transformer can only pay attention to past tokens. </p>\n": "<p>\u6b21\u306e\u30de\u30b9\u30af\u3092\u4f5c\u6210\u3057\u3066\u3001\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u304c\u904e\u53bb\u306e\u30c8\u30fc\u30af\u30f3\u306b\u3057\u304b\u6ce8\u76ee\u3067\u304d\u306a\u3044\u3088\u3046\u306b\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Create trainer </p>\n": "<p>\u30c8\u30ec\u30fc\u30ca\u30fc\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Cross-entropy loss </p>\n": "<p>\u30af\u30ed\u30b9\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u640d\u5931</p>\n",
|
||||
"<p>Data in the form of a tensor of ids </p>\n": "<p>id \u306e\u30c6\u30f3\u30bd\u30eb\u306e\u5f62\u5f0f\u306e\u30c7\u30fc\u30bf</p>\n",
|
||||
"<p>Download the file </p>\n": "<p>\u30d5\u30a1\u30a4\u30eb\u3092\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3059\u308b</p>\n",
|
||||
"<p>Embed the tokens (<span translate=no>_^_0_^_</span>) and run it through the the transformer </p>\n": "<p>\u30c8\u30fc\u30af\u30f3 (<span translate=no>_^_0_^_</span>) \u3092\u57cb\u3081\u8fbc\u307f\u3001\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306b\u901a\u3057\u307e\u3059</p>\n",
|
||||
"<p>Evaluate the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u8a55\u4fa1</p>\n",
|
||||
"<p>Extract the characters </p>\n": "<p>\u6587\u5b57\u3092\u62bd\u51fa</p>\n",
|
||||
"<p>FFN with Bilinear hidden layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30a4\u30ea\u30cb\u30a2\u96a0\u308c\u5c64\u4ed8\u304dFFN <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with GELU gate <span translate=no>_^_0_^_</span> </p>\n": "<p>GELU \u30b2\u30fc\u30c8\u4ed8\u304dFFN <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with Gated Linear Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b2\u30fc\u30c8\u4ed8\u304d\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8\u4ed8\u304dFFN <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with ReLU activation <span translate=no>_^_0_^_</span> </p>\n": "<p>RelU \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u3092\u4f7f\u7528\u3057\u305f FFN <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with ReLU gate <span translate=no>_^_0_^_</span> </p>\n": "<p>RelU \u30b2\u30fc\u30c8\u4ed8\u304d FN <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with Swish gate <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> </p>\n": "<p>\u30b9\u30a6\u30a3\u30c3\u30b7\u30e5\u30b2\u30fc\u30c8\u4ed8\u304d\u306eFFN\u3069\u3053 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Generate a sample </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u3092\u751f\u6210</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210</p>\n",
|
||||
"<p>Get the device </p>\n": "<p>\u30c7\u30d0\u30a4\u30b9\u3092\u5165\u624b</p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u30e2\u30c7\u30eb\u4e88\u6e2c\u3092\u53d6\u5f97 (\u6b32\u5f35\u308a)</p>\n",
|
||||
"<p>Gradient clipping norm </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u30ce\u30eb\u30e0</p>\n",
|
||||
"<p>Id to character map </p>\n": "<p>ID \u3092\u6587\u5b57\u30de\u30c3\u30d7\u306b\u5909\u63db</p>\n",
|
||||
"<p>Initialize <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u521d\u671f\u5316</p>\n",
|
||||
"<p>Initialize <a href=\"../mha.html\">Multi-Head Attention module</a> </p>\n": "<p><a href=\"../mha.html\">\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u521d\u671f\u5316</a></p>\n",
|
||||
"<p>Initialize the <a href=\"../models.html#TransformerLayer\">Transformer Block</a> </p>\n": "<p><a href=\"../models.html#TransformerLayer\">\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30d6\u30ed\u30c3\u30af\u3092\u521d\u671f\u5316</a></p>\n",
|
||||
"<p>Initialize the dataloader </p>\n": "<p>\u30c7\u30fc\u30bf\u30ed\u30fc\u30c0\u30fc\u306e\u521d\u671f\u5316</p>\n",
|
||||
"<p>Initialize the dataset </p>\n": "<p>\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u521d\u671f\u5316</p>\n",
|
||||
"<p>Initialize the model with an <a href=\"../models.html#EmbeddingsWithPositionalEncoding\">embedding layer</a> (with fixed positional encoding) <a href=\"../models.html#Encoder\">transformer encoder</a> and a linear layer to generate logits. </p>\n": "<p><a href=\"../models.html#EmbeddingsWithPositionalEncoding\">\u30e2\u30c7\u30eb\u3092\u57cb\u3081\u8fbc\u307f\u5c64 (\u56fa\u5b9a\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0</a>) <a href=\"../models.html#Encoder\">\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u3068\u7dda\u5f62\u5c64\u3067\u521d\u671f\u5316\u3057</a>\u3001\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Iterate over the minibatches </p>\n": "<p>\u30df\u30cb\u30d0\u30c3\u30c1\u3092\u53cd\u5fa9\u51e6\u7406</p>\n",
|
||||
"<p>Length of a training sample </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b5\u30f3\u30d7\u30eb\u306e\u9577\u3055</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Location of the text file </p>\n": "<p>\u30c6\u30ad\u30b9\u30c8\u30d5\u30a1\u30a4\u30eb\u306e\u5834\u6240</p>\n",
|
||||
"<p>Log the loss </p>\n": "<p>\u640d\u5931\u3092\u8a18\u9332\u3059\u308b</p>\n",
|
||||
"<p>Log the model parameters and gradients </p>\n": "<p>\u30e2\u30c7\u30eb\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3068\u52fe\u914d\u3092\u30ed\u30b0\u306b\u8a18\u9332\u3057\u307e\u3059</p>\n",
|
||||
"<p>Loop for the given number of epochs </p>\n": "<p>\u6307\u5b9a\u3055\u308c\u305f\u30a8\u30dd\u30c3\u30af\u6570\u306e\u30eb\u30fc\u30d7</p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u30c7\u30fc\u30bf\u3092\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Move the model to the current device </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u73fe\u5728\u306e\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Next token generation layer; this gives logits of the the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u751f\u6210\u30ec\u30a4\u30e4\u30fc\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u304c\u5f97\u3089\u308c\u307e\u3059</p>\n",
|
||||
"<p>Number of different characters </p>\n": "<p>\u7570\u306a\u308b\u6587\u5b57\u306e\u6570</p>\n",
|
||||
"<p>Number of training epochs; <em>note that our dataset definition repeats the data <span translate=no>_^_0_^_</span> times in a single epoch</em> </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30a8\u30dd\u30c3\u30af\u306e\u6570\u3002<em>\u3053\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u5b9a\u7fa9\u3067\u306f\u3001<span translate=no>_^_0_^_</span> 1\u3064\u306e\u30a8\u30dd\u30c3\u30af\u3067\u30c7\u30fc\u30bf\u56de\u6570\u304c\u7e70\u308a\u8fd4\u3055\u308c\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044</em></p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u51fa\u529b\u3092\u5370\u5237\u3059\u308b</p>\n",
|
||||
"<p>Read the downloaded file </p>\n": "<p>\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3057\u305f\u30d5\u30a1\u30a4\u30eb\u3092\u8aad\u307f\u8fbc\u3080</p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>25\u30c8\u30fc\u30af\u30f3\u306e\u30b5\u30f3\u30d7\u30eb</p>\n",
|
||||
"<p>Save the model </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u4fdd\u5b58\u3059\u308b</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u8ffd\u8de1\u3057\u305f\u30e1\u30c8\u30ea\u30af\u30b9\u3092\u4fdd\u5b58\u3059\u308b</p>\n",
|
||||
"<p>Set model state to training </p>\n": "<p>\u30e2\u30c7\u30eb\u72b6\u614b\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306b\u8a2d\u5b9a</p>\n",
|
||||
"<p>Set models for training and loading </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u30ed\u30fc\u30c9\u7528\u306e\u30e2\u30c7\u30eb\u306e\u8a2d\u5b9a</p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u30c8\u30e9\u30c3\u30ab\u30fc\u69cb\u6210\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Set tracker step, as the number of characters trained on </p>\n": "<p>\u30c8\u30e9\u30c3\u30ab\u30fc\u30b9\u30c6\u30c3\u30d7\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u305f\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u306e\u6570\u3068\u3057\u3066\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u8d77\u52d5\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u6700\u9069\u5316\u306e\u4e00\u6b69\u3092\u8e0f\u307f\u51fa\u3059</p>\n",
|
||||
"<p>This will be initialized on the first call </p>\n": "<p>\u3053\u308c\u306f\u6700\u521d\u306e\u547c\u3073\u51fa\u3057\u3067\u521d\u671f\u5316\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u3092\u30c8\u30fc\u30af\u30f3\u5316</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Transformer based encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30d9\u30fc\u30b9\u306e\u30a8\u30f3\u30b3\u30fc\u30c0</p>\n",
|
||||
"Gated Linear Units and Variants": "\u30b2\u30fc\u30c6\u30c3\u30c9\u30ea\u30cb\u30a2\u30e6\u30cb\u30c3\u30c8\u3068\u30d0\u30ea\u30a2\u30f3\u30c8",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u4f4d\u7f6e\u5358\u4f4d\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af (FFN) \u306e\u30b2\u30fc\u30c8\u7dda\u5f62\u5358\u4f4d\u3068\u30d0\u30ea\u30a2\u30f3\u30c8\u3092\u4f7f\u7528\u3057\u3066\u81ea\u5df1\u56de\u5e30\u5909\u5727\u5668\u306b\u5b66\u7fd2\u3055\u305b\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<p>This trains a simple <a href=\"../../\">transformer</a> model for auto-regression. We try different variants for the <a href=\"../feed_forward\">position-wise feedforward network</a>.</p>\n<p><em>This is a simpler implementation that doesn't use <a href=\"experiment.html\"><span translate=no>_^_0_^_</span></a> module. We decided to write a simpler implementation to make it easier for readers who are not familiar.</em></p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/glu_variants/simple.ipynb\"><span translate=no>_^_1_^_</span></a> <a href=\"https://app.labml.ai/run/86b773f65fc911ebb2ac0242ac1c0002\"><span translate=no>_^_2_^_</span></a></p>\n": "<h1>\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf</h1>\n</a> <p>\u0db8\u0dd9\u0dba\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dbb\u0dbd <a href=\"../../\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2. <a href=\"../feed_forward\">\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0ddd\u0dc2\u0d9a \u0da2\u0dcf\u0dbd\u0dba</a>\u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 \u0dc0\u0dd2\u0dc0\u0dd2\u0db0 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf \u0d8b\u0dad\u0dca\u0dc3\u0dcf\u0dc4 \u0d9a\u0dbb\u0db8\u0dd4. </p>\n<p><em>\u0db8\u0dd9\u0dba <a href=\"experiment.html\"><span translate=no>_^_0_^_</span></a> \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0db1 \u0dc3\u0dbb\u0dbd \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2. \u0dc4\u0dd4\u0dbb\u0dd4\u0db4\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0db1\u0ddc\u0dc0\u0db1 \u0db4\u0dcf\u0da8\u0d9a\u0dba\u0db1\u0dca\u0da7 \u0db4\u0dc4\u0dc3\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dbb\u0dbd \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dca \u0dbd\u0dd2\u0dc0\u0dd3\u0db8\u0da7 \u0d85\u0db4\u0dd2 \u0dad\u0dd3\u0dbb\u0dab\u0dba \u0d9a\u0dc5\u0dcf. </em></p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/glu_variants/simple.ipynb\"><span translate=no>_^_1_^_</span></a> <a href=\"https://app.labml.ai/run/86b773f65fc911ebb2ac0242ac1c0002\"> <span translate=no>_^_2_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Trainer</h2>\n": "<h2>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4</h2>\n",
|
||||
"<h3>Configurations</h3>\n": "<h3>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0dc0\u0dbb\u0dd2\u0db1\u0dca \u0dc0\u0dbb \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba</h3>\n",
|
||||
"<h3>Tiny Shakespeare Dataset</h3>\n": "<h3>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba</h3>\n",
|
||||
"<h3>Train the model</h3>\n": "<h3>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<p> Number of samples in the dataset.</p>\n<p><em>This will read the dataset <span translate=no>_^_0_^_</span> times in a single epoch.</em></p>\n": "<p> \u0daf\u0dad\u0dca\u0dad\u0d9a\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0dda \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0d9c\u0dab\u0db1. </p>\n<p><em>\u0db8\u0dd9\u0dba\u0dad\u0db1\u0dd2 \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd <span translate=no>_^_0_^_</span> \u0dc0\u0dda\u0dbd\u0dcf\u0dc0\u0db1\u0dca \u0d9a\u0dd2\u0dba\u0dc0\u0db1\u0dd4 \u0d87\u0dad. </em></p>\n",
|
||||
"<p> Return a sample</p>\n": "<p> \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba\u0d9a\u0dca\u0d86\u0db4\u0dc3\u0dd4 \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p> Transform the text into a tensor of ids</p>\n": "<p> \u0db4\u0dd9\u0dc5\u0dc4\u0dd0\u0db3\u0dd4\u0db1\u0dd4\u0db8\u0dca\u0db4\u0dad\u0dca\u0dc0\u0dbd \u0d86\u0dad\u0dad\u0dd2\u0dba\u0d9a\u0da7 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u0dbd\u0ddc\u0d9c\u0dca\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0da7\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate loss </p>\n": "<p>\u0d85\u0dbd\u0dcf\u0db7\u0dba\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Character to id (integer) map </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0dba(\u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf) \u0dc3\u0dd2\u0dad\u0dd2\u0dba\u0db8\u0da7 </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d89\u0dc0\u0dad\u0dca </p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dca\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a </p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create subsequent mask, so that the transformer can only pay attention to past tokens. </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d85\u0dad\u0dd3\u0dad \u0da7\u0ddd\u0d9a\u0db1 \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0db4\u0db8\u0dab\u0d9a\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0dc0\u0db1 \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Create trainer </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Cross-entropy loss </p>\n": "<p>\u0dc4\u0dbb\u0dc3\u0dca\u0d91\u0db1\u0dca\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd2 \u0db1\u0dd0\u0dad\u0dd2\u0dc0\u0dd3\u0db8 </p>\n",
|
||||
"<p>Data in the form of a tensor of ids </p>\n": "<p>\u0dc4\u0dd0\u0db3\u0dd4\u0db1\u0dd4\u0db8\u0dca\u0db4\u0dad\u0dca\u0d86\u0dad\u0dad\u0dd2\u0dba\u0d9a \u0dc3\u0dca\u0dc0\u0dbb\u0dd6\u0db4\u0dba\u0dd9\u0db1\u0dca \u0daf\u0dad\u0dca\u0dad </p>\n",
|
||||
"<p>Download the file </p>\n": "<p>\u0d9c\u0ddc\u0db1\u0dd4\u0dc0\u0db6\u0dcf\u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Embed the tokens (<span translate=no>_^_0_^_</span>) and run it through the the transformer </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 (<span translate=no>_^_0_^_</span>) \u0dc3\u0dc4 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0d91\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Evaluate the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dad\u0d9a\u0dca\u0dc3\u0dda\u0dbb\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Extract the characters </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0d8b\u0db4\u0dd4\u0da7\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>FFN with Bilinear hidden layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db6\u0dd2\u0dbd\u0dd3\u0db1\u0dd2\u0dba\u0dbb\u0dca\u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dad\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0d9a\u0dca \u0dc3\u0dc4\u0dd2\u0dad FFN <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>FFN with GELU gate <span translate=no>_^_0_^_</span> </p>\n": "<p>GELU\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dc0 \u0dc3\u0dc4\u0dd2\u0dad FFN <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>FFN with Gated Linear Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a\u0dba \u0dc3\u0db8\u0d9f FFN <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>FFN with ReLU activation <span translate=no>_^_0_^_</span> </p>\n": "<p>RelU\u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db8\u0d9f FFN <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>FFN with ReLU gate <span translate=no>_^_0_^_</span> </p>\n": "<p>RelU\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dc0 \u0dc3\u0db8\u0d9f FFN <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>FFN with Swish gate <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0dca\u0dc0\u0dd2\u0dc2\u0dca <span translate=no>_^_0_^_</span> \u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4\u0dc0 \u0dc3\u0db8\u0d9f \u0d91\u0dc6\u0dca\u0d91\u0dc6\u0dca\u0d91\u0db1\u0dca <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Generate a sample </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba\u0d9a\u0dca\u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 (\u0d9a\u0dd1\u0daf\u0dbb) </p>\n",
|
||||
"<p>Gradient clipping norm </p>\n": "<p>\u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba </p>\n",
|
||||
"<p>Id to character map </p>\n": "<p>Id\u0dc3\u0dd2\u0da7 \u0d85\u0d9a\u0dca\u0dc2\u0dbb \u0dc3\u0dd2\u0dad\u0dd2\u0dba\u0db8 </p>\n",
|
||||
"<p>Initialize <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize <a href=\"../mha.html\">Multi-Head Attention module</a> </p>\n": "<p><a href=\"../mha.html\">\u0db6\u0dc4\u0dd4-\u0dc4\u0dd2\u0dc3 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba</a> \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize the <a href=\"../models.html#TransformerLayer\">Transformer Block</a> </p>\n": "<p><a href=\"../models.html#TransformerLayer\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0ddc\u0da7\u0dc3</a> \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize the dataloader </p>\n": "<p>\u0daf\u0dad\u0dca\u0dad\u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize the dataset </p>\n": "<p>\u0daf\u0dad\u0dca\u0dad\u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize the model with an <a href=\"../models.html#EmbeddingsWithPositionalEncoding\">embedding layer</a> (with fixed positional encoding) <a href=\"../models.html#Encoder\">transformer encoder</a> and a linear layer to generate logits. </p>\n": "<p>\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca\u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf <a href=\"../models.html#EmbeddingsWithPositionalEncoding\">\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dca</a> (\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dc4\u0dd2\u0dad) <a href=\"../models.html#Encoder\">\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a> \u0dc3\u0dc4 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dad\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0d9a\u0dca \u0dc3\u0dc4\u0dd2\u0dad \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Iterate over the minibatches </p>\n": "<p>\u0db8\u0dd2\u0db1\u0dd2\u0db6\u0dd0\u0da0\u0dca\u0dc4\u0dbb\u0dc4\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Length of a training sample </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba\u0d9a \u0daf\u0dd2\u0d9c </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Location of the text file </p>\n": "<p>\u0db4\u0dd9\u0dc5\u0d9c\u0ddc\u0db1\u0dd4\u0dc0\u0dda \u0db4\u0dd2\u0dc4\u0dd2\u0da7\u0dd3\u0db8 </p>\n",
|
||||
"<p>Log the loss </p>\n": "<p>\u0d85\u0dbd\u0dcf\u0db7\u0dba\u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the model parameters and gradients </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0dc4 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Loop for the given number of epochs </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0daf\u0dd3 \u0d87\u0dad\u0dd2 \u0d91\u0db4\u0ddc\u0da0\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0dbd\u0dd6\u0db4\u0dca </p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dc0\u0dd9\u0dad \u0daf\u0dad\u0dca\u0dad \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move the model to the current device </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca \u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0da7 \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Next token generation layer; this gives logits of the the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1 \u0dc3\u0dca\u0dad\u0dbb\u0dba; \u0db8\u0dd9\u0dba \u0d8a\u0dc5\u0d9f \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0dba\u0dd2 </p>\n",
|
||||
"<p>Number of different characters </p>\n": "<p>\u0dc0\u0dd2\u0dc0\u0dd2\u0db0\u0d85\u0d9a\u0dca\u0dc2\u0dbb \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of training epochs; <em>note that our dataset definition repeats the data <span translate=no>_^_0_^_</span> times in a single epoch</em> </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 <em>\u0d91\u0db4\u0ddc\u0da0\u0dca \u0d9c\u0dab\u0db1; \u0d85\u0db4\u0d9c\u0dda \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd \u0d85\u0dbb\u0dca\u0dae \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd3\u0db8 \u0dad\u0db1\u0dd2 \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0daf\u0dad\u0dca\u0dad <span translate=no>_^_0_^_</span> \u0dc0\u0dda\u0dbd\u0dcf\u0dc0\u0db1\u0dca \u0db4\u0dd4\u0db1\u0dbb\u0dcf\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1</em> </p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Read the downloaded file </p>\n": "<p>\u0db6\u0dcf\u0d9c\u0dad\u0d9a\u0dc5 \u0d9c\u0ddc\u0db1\u0dd4\u0dc0 \u0d9a\u0dd2\u0dba\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd25 \u0da7\u0ddd\u0d9a\u0db1 </p>\n",
|
||||
"<p>Save the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u0dbd\u0dd4\u0dc4\u0dd4\u0db6\u0dd0\u0db3\u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set model state to training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0dad\u0dad\u0dca\u0dc0\u0dba \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for training and loading </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dd0\u0d9a\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set tracker step, as the number of characters trained on </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d85\u0d9a\u0dca\u0dc2\u0dbb \u0d9c\u0dab\u0db1 \u0dbd\u0dd9\u0dc3 \u0da7\u0dca\u0dbb\u0dd0\u0d9a\u0dbb\u0dca \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>This will be initialized on the first call </p>\n": "<p>\u0db8\u0dd9\u0dba\u0db4\u0dc5\u0db8\u0dd4 \u0d87\u0db8\u0dad\u0dd4\u0db8\u0dd9\u0db1\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd9\u0dbb\u0dda </p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0da7\u0ddd\u0d9a\u0dd9\u0db1\u0dca\u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Transformer based encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"Gated Linear Units and Variants": "\u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0ddd\u0dc2\u0d9a \u0da2\u0dcf\u0dbd\u0dba (FFN) \u0dc3\u0db3\u0dc4\u0dcf \u0d9c\u0dda\u0da7\u0dca\u0da7\u0dd4 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d92\u0d9a\u0d9a \u0dc3\u0dc4 \u0db4\u0dca\u0dbb\u0db7\u0dda\u0daf \u0dc3\u0db8\u0d9f \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1."
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"<h1>Gated Linear Units and Variants</h1>\n<p>This trains a simple <a href=\"../../\">transformer</a> model for auto-regression. We try different variants for the <a href=\"../feed_forward\">position-wise feedforward network</a>.</p>\n<p><em>This is a simpler implementation that doesn't use <a href=\"experiment.html\"><span translate=no>_^_0_^_</span></a> module. We decided to write a simpler implementation to make it easier for readers who are not familiar.</em></p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/glu_variants/simple.ipynb\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u95e8\u63a7\u7ebf\u6027\u5355\u5143\u548c\u53d8\u4f53</h1>\n</a><p>\u8fd9\u53ef\u4ee5\u8bad\u7ec3\u4e00\u4e2a\u7b80\u5355\u7684<a href=\"../../\">\u53d8\u538b\u5668\u6a21\u578b\u8fdb\u884c\u81ea\u52a8\u56de\u5f52\u3002\u6211\u4eec\u4e3a<a href=\"../feed_forward\">\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc</a>\u5c1d\u8bd5\u4e0d\u540c\u7684\u53d8\u4f53\u3002</p>\n<p><em>\u8fd9\u662f\u4e00\u4e2a\u4e0d\u4f7f\u7528<a href=\"experiment.html\"><span translate=no>_^_0_^_</span></a>\u6a21\u5757\u7684\u66f4\u7b80\u5355\u7684\u5b9e\u73b0\u3002\u6211\u4eec\u51b3\u5b9a\u7f16\u5199\u4e00\u4e2a\u66f4\u7b80\u5355\u7684\u5b9e\u73b0\uff0c\u8ba9\u4e0d\u719f\u6089\u7684\u8bfb\u8005\u66f4\u5bb9\u6613\u4f7f\u7528\u3002</em></p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/glu_variants/simple.ipynb\"><span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52a8\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Trainer</h2>\n": "<h2>\u8bad\u7ec3\u5e08</h2>\n",
|
||||
"<h3>Configurations</h3>\n": "<h3>\u914d\u7f6e</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u91c7\u6837\u529f\u80fd\u53ef\u5728\u8bad\u7ec3\u65f6\u5b9a\u671f\u751f\u6210\u6837\u672c</h3>\n",
|
||||
"<h3>Tiny Shakespeare Dataset</h3>\n": "<h3>\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</h3>\n",
|
||||
"<h3>Train the model</h3>\n": "<h3>\u8bad\u7ec3\u6a21\u578b</h3>\n",
|
||||
"<p> Number of samples in the dataset.</p>\n<p><em>This will read the dataset <span translate=no>_^_0_^_</span> times in a single epoch.</em></p>\n": "<p>\u6570\u636e\u96c6\u4e2d\u7684\u6837\u672c\u6570\u3002</p>\n<p><em>\u8fd9\u5c06\u8bfb\u53d6\u5355\u4e2a\u7eaa\u5143\u4e2d\u7684\u6570\u636e\u96c6<span translate=no>_^_0_^_</span>\u65f6\u95f4\u3002</em></p>\n",
|
||||
"<p> Return a sample</p>\n": "<p>\u8fd4\u56de\u6837\u54c1</p>\n",
|
||||
"<p> Transform the text into a tensor of ids</p>\n": "<p>\u5c06\u6587\u672c\u8f6c\u6362\u4e3a id \u5f20\u91cf</p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u6dfb\u52a0\u65e5\u5fd7\u8bb0\u5f55\u7684\u9884\u6d4b</p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u5c06\u9884\u6d4b\u6dfb\u52a0\u5230\u63d0\u793a\u7b26\u4e2d</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"<p>Calculate loss </p>\n": "<p>\u8ba1\u7b97\u635f\u5931</p>\n",
|
||||
"<p>Character to id (integer) map </p>\n": "<p>\u5b57\u7b26\u5230 id\uff08\u6574\u6570\uff09\u6620\u5c04</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u6e05\u9664\u6e10\u53d8</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u526a\u8f91\u6e10\u53d8</p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u6536\u96c6\u8f93\u51fa\u4ee5\u8fdb\u884c\u6253\u5370</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Create subsequent mask, so that the transformer can only pay attention to past tokens. </p>\n": "<p>\u521b\u5efa\u540e\u7eed\u63a9\u7801\uff0c\u4ee5\u4fbf\u53d8\u538b\u5668\u53ea\u80fd\u5173\u6ce8\u8fc7\u53bb\u7684\u4ee4\u724c\u3002</p>\n",
|
||||
"<p>Create trainer </p>\n": "<p>\u521b\u5efa\u8bad\u7ec3\u5668</p>\n",
|
||||
"<p>Cross-entropy loss </p>\n": "<p>\u4ea4\u53c9\u71b5\u635f\u5931</p>\n",
|
||||
"<p>Data in the form of a tensor of ids </p>\n": "<p>\u4ee5 id \u5f20\u91cf\u5f62\u5f0f\u663e\u793a\u7684\u6570\u636e</p>\n",
|
||||
"<p>Download the file </p>\n": "<p>\u4e0b\u8f7d\u8be5\u6587\u4ef6</p>\n",
|
||||
"<p>Embed the tokens (<span translate=no>_^_0_^_</span>) and run it through the the transformer </p>\n": "<p>\u5d4c\u5165\u4ee4\u724c (<span translate=no>_^_0_^_</span>) \u5e76\u901a\u8fc7\u53d8\u538b\u5668\u8fd0\u884c\u5b83</p>\n",
|
||||
"<p>Evaluate the model </p>\n": "<p>\u8bc4\u4f30\u6a21\u578b</p>\n",
|
||||
"<p>Extract the characters </p>\n": "<p>\u63d0\u53d6\u5b57\u7b26</p>\n",
|
||||
"<p>FFN with Bilinear hidden layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e26\u53cc\u7ebf\u6027\u9690\u85cf\u5c42\u7684 FFN<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with GELU gate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e26\u6709 GELU \u95e8\u7684 FFN<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with Gated Linear Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e26\u95e8\u63a7\u7ebf\u6027\u5355\u5143\u7684 FFN<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with ReLU activation <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6fc0\u6d3b ReLU \u7684 FFN<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with ReLU gate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e26\u6709 ReLU \u95e8\u7684 FFN<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>FFN with Swish gate <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> </p>\n": "<p>FFN \u6709 Swish gate<span translate=no>_^_0_^_</span> \u5728\u54ea\u91cc<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Generate a sample </p>\n": "<p>\u751f\u6210\u6837\u672c</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u751f\u6210\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Get the device </p>\n": "<p>\u62ff\u5230\u8bbe\u5907</p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u9884\u6d4b\uff08\u8d2a\u5a6a\uff09</p>\n",
|
||||
"<p>Gradient clipping norm </p>\n": "<p>\u6e10\u53d8\u526a\u5207\u89c4\u8303</p>\n",
|
||||
"<p>Id to character map </p>\n": "<p>\u89d2\u8272\u6620\u5c04\u7684 ID</p>\n",
|
||||
"<p>Initialize <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u521d\u59cb\u5316 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Initialize <a href=\"../mha.html\">Multi-Head Attention module</a> </p>\n": "<p>\u521d\u59cb\u5316<a href=\"../mha.html\">\u591a\u5934\u6ce8\u610f\u6a21\u5757</a></p>\n",
|
||||
"<p>Initialize the <a href=\"../models.html#TransformerLayer\">Transformer Block</a> </p>\n": "<p>\u521d\u59cb\u5316\u53d8<a href=\"../models.html#TransformerLayer\">\u538b\u5668\u6a21\u5757</a></p>\n",
|
||||
"<p>Initialize the dataloader </p>\n": "<p>\u521d\u59cb\u5316\u6570\u636e\u52a0\u8f7d\u5668</p>\n",
|
||||
"<p>Initialize the dataset </p>\n": "<p>\u521d\u59cb\u5316\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Initialize the model with an <a href=\"../models.html#EmbeddingsWithPositionalEncoding\">embedding layer</a> (with fixed positional encoding) <a href=\"../models.html#Encoder\">transformer encoder</a> and a linear layer to generate logits. </p>\n": "\u4f7f\u7528@@ <p><a href=\"../models.html#EmbeddingsWithPositionalEncoding\">\u5d4c\u5165\u5c42</a>\uff08\u5177\u6709\u56fa\u5b9a\u4f4d\u7f6e\u7f16\u7801\uff09<a href=\"../models.html#Encoder\">\u53d8\u538b\u5668\u7f16\u7801\u5668\u548c</a>\u7ebf\u6027\u5c42\u6765\u521d\u59cb\u5316\u6a21\u578b\u4ee5\u751f\u6210\u5bf9\u6570\u3002</p>\n",
|
||||
"<p>Iterate over the minibatches </p>\n": "<p>\u904d\u5386\u8ff7\u4f60\u6279\u6b21</p>\n",
|
||||
"<p>Length of a training sample </p>\n": "<p>\u8bad\u7ec3\u6837\u672c\u7684\u957f\u5ea6</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Location of the text file </p>\n": "<p>\u6587\u672c\u6587\u4ef6\u7684\u4f4d\u7f6e</p>\n",
|
||||
"<p>Log the loss </p>\n": "<p>\u8bb0\u5f55\u635f\u5931</p>\n",
|
||||
"<p>Log the model parameters and gradients </p>\n": "<p>\u8bb0\u5f55\u6a21\u578b\u53c2\u6570\u548c\u68af\u5ea6</p>\n",
|
||||
"<p>Loop for the given number of epochs </p>\n": "<p>\u5faa\u73af\u4f7f\u7528\u7ed9\u5b9a\u6570\u91cf\u7684\u5468\u671f</p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u5c06\u6570\u636e\u79fb\u52a8\u5230\u8bbe\u5907</p>\n",
|
||||
"<p>Move the model to the current device </p>\n": "<p>\u5c06\u6a21\u578b\u79fb\u81f3\u5f53\u524d\u8bbe\u5907</p>\n",
|
||||
"<p>Next token generation layer; this gives logits of the the next token </p>\n": "<p>\u4e0b\u4e00\u4ee3\u5e01\u751f\u6210\u5c42\uff1b\u8fd9\u7ed9\u51fa\u4e86\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Number of different characters </p>\n": "<p>\u4e0d\u540c\u5b57\u7b26\u7684\u6570\u91cf</p>\n",
|
||||
"<p>Number of training epochs; <em>note that our dataset definition repeats the data <span translate=no>_^_0_^_</span> times in a single epoch</em> </p>\n": "<p>\u8bad\u7ec3\u5468\u671f\u7684\u6570\u91cf\uff1b<em>\u8bf7\u6ce8\u610f\uff0c\u6211\u4eec\u7684\u6570\u636e\u96c6\u5b9a\u4e49\u5728\u5355\u4e2a\u7eaa\u5143\u4e2d\u91cd\u590d\u6570\u636e<span translate=no>_^_0_^_</span>\u65f6\u95f4</em></p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u6253\u5370\u91c7\u6837\u8f93\u51fa</p>\n",
|
||||
"<p>Read the downloaded file </p>\n": "<p>\u8bfb\u53d6\u4e0b\u8f7d\u7684\u6587\u4ef6</p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>\u6837\u672c 25 \u4e2a\u4ee3\u5e01</p>\n",
|
||||
"<p>Save the model </p>\n": "<p>\u4fdd\u5b58\u6a21\u578b</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u4fdd\u5b58\u8ddf\u8e2a\u7684\u6307\u6807</p>\n",
|
||||
"<p>Set model state to training </p>\n": "<p>\u5c06\u6a21\u578b\u72b6\u6001\u8bbe\u7f6e\u4e3a\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for training and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u8bad\u7ec3\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u8bbe\u7f6e\u8ddf\u8e2a\u5668\u914d\u7f6e</p>\n",
|
||||
"<p>Set tracker step, as the number of characters trained on </p>\n": "<p>\u5c06\u8ddf\u8e2a\u5668\u6b65\u957f\u8bbe\u7f6e\u4e3a\u8bad\u7ec3\u7684\u5b57\u7b26\u6570</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u542f\u52a8\u63d0\u793a</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u91c7\u53d6\u4f18\u5316\u5668\u6b65\u9aa4</p>\n",
|
||||
"<p>This will be initialized on the first call </p>\n": "<p>\u8fd9\u5c06\u5728\u7b2c\u4e00\u6b21\u8c03\u7528\u65f6\u521d\u59cb\u5316</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u6a21\u5757</p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u5c06\u63d0\u793a\u7b26\u53f7\u5316</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u8bad\u7ec3\u6a21\u578b</p>\n",
|
||||
"<p>Transformer based encoder </p>\n": "<p>\u57fa\u4e8e\u53d8\u538b\u5668\u7684\u7f16\u7801\u5668</p>\n",
|
||||
"Gated Linear Units and Variants": "\u95e8\u63a7\u7ebf\u6027\u5355\u4f4d\u548c\u53d8\u4f53",
|
||||
"Train an auto-regressive transformer with Gated Linear Units and variants for the position-wise feedforward network (FFN).": "\u4f7f\u7528\u95e8\u63a7\u7ebf\u6027\u5355\u5143\u548c\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc (FFN) \u53d8\u4f53\u8bad\u7ec3\u81ea\u56de\u5f52\u53d8\u538b\u5668\u3002"
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"<h1>Pay Attention to MLPs (gMLP)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a gMLP model based autoregressive model.</p>\n": "<h1>MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044</h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.08050\">MLP\u306b\u6ce8\u610f\u3057\u3066</a>\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a>\u3002</p>\n<p><strong>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30b2\u30fc\u30c6\u30a3\u30f3\u30b0\u3092\u5099\u3048\u305f\u591a\u5c64\u30d1\u30fc\u30bb\u30d7\u30c8\u30ed\u30f3\uff08MLP\uff09\u30d9\u30fc\u30b9\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\uff08GMLP\u3068\u540d\u4ed8\u3051\u3089\u308c\u3066\u3044\u307e\u3059\uff09\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002</strong><span translate=no>_^_0_^_</span><em>gMLP</em> \u30d6\u30ed\u30c3\u30af\u306e\u30b9\u30bf\u30c3\u30af\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059</p>\u3002\n<p><a href=\"experiment.html\">gMLP\u30e2\u30c7\u30eb\u30d9\u30fc\u30b9\u306e\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059</a>\u3002</p>\n",
|
||||
"<h2>Spatial Gating Unit</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a linear transformation along the sequence dimension, and <span translate=no>_^_2_^_</span> is element-wise multiplication. <span translate=no>_^_3_^_</span> is split into to parts of equal size <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> along the channel dimension (embedding dimension).</p>\n": "<h2>\u7a7a\u9593\u30b2\u30fc\u30c8\u30e6\u30cb\u30c3\u30c8</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>\u3053\u3053\u3067<span translate=no>_^_1_^_</span>\u3001\u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u6b21\u5143\u306b\u6cbf\u3063\u305f\u7dda\u5f62\u5909\u63db\u3067\u3001<span translate=no>_^_2_^_</span>\u306f\u8981\u7d20\u5358\u4f4d\u306e\u4e57\u7b97\u3067\u3059\u3002<span translate=no>_^_3_^_</span>\u30c1\u30e3\u30cd\u30eb\u5bf8\u6cd5\uff08\u57cb\u3081\u8fbc\u307f\u5bf8\u6cd5\uff09<span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span>\u306b\u6cbf\u3063\u3066\u540c\u3058\u30b5\u30a4\u30ba\u306e2\u3064\u306e\u90e8\u5206\u306b\u5206\u5272\u3055\u308c\u307e\u3059</p>\u3002\n",
|
||||
"<h2>gMLP Block</h2>\n<p>Each block does the following transformations to input embeddings <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the sequence length and <span translate=no>_^_2_^_</span> is the dimensionality of the embeddings:</p>\n<span translate=no>_^_3_^_</span><p>where <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are learnable projection weights. <span translate=no>_^_6_^_</span> is the Spacial Gating Unit defined below. Output dimensionality of <span translate=no>_^_7_^_</span> will be half of <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span> is an activation function such as <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLU</a>.</p>\n": "<h2>GmLP \u30d6\u30ed\u30c3\u30af</h2>\n<p>\u5404\u30d6\u30ed\u30c3\u30af\u306f\u3001\u5165\u529b\u57cb\u3081\u8fbc\u307f\u306b\u5bfe\u3057\u3066\u6b21\u306e\u5909\u63db\u3092\u884c\u3044\u307e\u3059\u3002<span translate=no>_^_0_^_</span>\u3053\u3053\u3067\u3001\u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u306f\u57cb\u3081\u8fbc\u307f\u306e\u6b21\u5143\u3067\u3059\u3002</p>\n<span translate=no>_^_3_^_</span><p><span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span>\u5b66\u7fd2\u53ef\u80fd\u306a\u6295\u5f71\u91cd\u307f\u306e\u4f4d\u7f6e\u3068\u4f4d\u7f6e<span translate=no>_^_6_^_</span>\u306f\u4ee5\u4e0b\u306b\u5b9a\u7fa9\u3059\u308b\u30b9\u30da\u30fc\u30b7\u30e3\u30eb\u30fb\u30b2\u30fc\u30c6\u30a3\u30f3\u30b0\u30fb\u30e6\u30cb\u30c3\u30c8\u3067\u3059\u3002<span translate=no>_^_7_^_</span>\u306e\u51fa\u529b\u6b21\u5143\u306f\u306e\u534a\u5206\u306b\u306a\u308a\u307e\u3059\u3002<span translate=no>_^_8_^_</span><span translate=no>_^_9_^_</span><a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">\u306fGelU\u306e\u3088\u3046\u306a\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u95a2\u6570\u3067\u3059</a></p>\u3002\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span>. The batch dimension should be of size <span translate=no>_^_2_^_</span> because this implementation supports only same mask for all samples in the batch. </p>\n": "<p><span translate=no>_^_0_^_</span>\u5f62\u304c\u3042\u308a\u307e\u3059<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u3053\u306e\u5b9f\u88c5\u3067\u306f\u30d0\u30c3\u30c1\u5185\u306e\u3059\u3079\u3066\u306e\u30b5\u30f3\u30d7\u30eb\u306b\u5bfe\u3057\u3066\u540c\u3058\u30de\u30b9\u30af\u3057\u304b\u30b5\u30dd\u30fc\u30c8\u3057\u306a\u3044\u305f\u3081\u3001\u30d0\u30c3\u30c1\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u306f\u30b5\u30a4\u30ba\u306b\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<p>Activation function <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u6a5f\u80fd <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Add the shortcut connection </p>\n": "<p>\u30b7\u30e7\u30fc\u30c8\u30ab\u30c3\u30c8\u63a5\u7d9a\u3092\u8ffd\u52a0\u3059\u308b</p>\n",
|
||||
"<p>Apply mask to the weights.</p>\n<p>If <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> then <span translate=no>_^_2_^_</span> will not get any information from token <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u30a6\u30a7\u30a4\u30c8\u306b\u30de\u30b9\u30af\u3092\u304b\u3051\u307e\u3059\u3002</p>\n<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3082\u3057\u305d\u3046\u306a\u3089<span translate=no>_^_2_^_</span>\u3001\u30c8\u30fc\u30af\u30f3\u304b\u3089\u60c5\u5831\u3092\u53d6\u5f97\u3059\u308b\u3053\u3068\u306f\u306a\u3044<span translate=no>_^_3_^_</span>\u3002</p>\n",
|
||||
"<p>Check mask </p>\n": "<p>\u30c1\u30a7\u30c3\u30af\u30de\u30b9\u30af</p>\n",
|
||||
"<p>Embedding size (required by <a href=\"../models.html#Encoder\">Encoder</a>. We use the encoder module from transformer architecture and plug <em>gMLP</em> block as a replacement for the <a href=\"../models.html#Encoder\">Transformer Layer</a>. </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba (<a href=\"../models.html#Encoder\">\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u5fc5\u8981</a>)</p><a href=\"../models.html#Encoder\">\u30c8\u30e9\u30f3\u30b9\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u30a8\u30f3\u30b3\u30fc\u30c0\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u4f7f\u7528\u3057\u3001<em>\u30c8\u30e9\u30f3\u30b9\u30ec\u30a4\u30e4\u306e\u4ee3\u308f\u308a\u306bgMLP\u30d6\u30ed\u30c3\u30af\u3092\u30d7\u30e9\u30b0\u3057\u307e\u3059</em>\u3002</a>\n",
|
||||
"<p>Final projection <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u7d42\u6295\u5f71 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get sequence length </p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the weight matrix; truncate if larger than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30a6\u30a7\u30a4\u30c8\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u3092\u53d6\u5f97\u3002\u3053\u308c\u3088\u308a\u5927\u304d\u3044\u5834\u5408\u306f\u5207\u308a\u6368\u3066\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Here we only support the same mask for all samples </p>\n": "<p>\u3053\u3053\u3067\u306f\u3001\u3059\u3079\u3066\u306e\u30b5\u30f3\u30d7\u30eb\u3067\u540c\u3058\u30de\u30b9\u30af\u306e\u307f\u3092\u30b5\u30dd\u30fc\u30c8\u3057\u3066\u3044\u307e\u3059</p>\n",
|
||||
"<p>Keep a copy for shortcut connection </p>\n": "<p>\u30b7\u30e7\u30fc\u30c8\u30ab\u30c3\u30c8\u63a5\u7d9a\u7528\u306b\u30b3\u30d4\u30fc\u3092\u4fdd\u5b58</p>\n",
|
||||
"<p>Normalization layer before applying <span translate=no>_^_0_^_</span> </p>\n": "<p>\u9069\u7528\u524d\u306e\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Normalization layer fro Pre-Norm </p>\n": "<p>\u30d7\u30ec\u30ce\u30eb\u30e0\u306e\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30ce\u30fc\u30de\u30e9\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> before <span translate=no>_^_1_^_</span> </p>\n": "<p>\u524d\u306b\u30ce\u30fc\u30de\u30e9\u30a4\u30ba <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Projection and activation <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d7\u30ed\u30b8\u30a7\u30af\u30b7\u30e7\u30f3\u3068\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Projection layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u306e\u6295\u5f71\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Remove the batch dimension </p>\n": "<p>\u30d0\u30c3\u30c1\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u3092\u524a\u9664\u3059\u308b</p>\n",
|
||||
"<p>Spacial Gating Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7a7a\u9593\u30b2\u30fc\u30c8\u30e6\u30cb\u30c3\u30c8 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Split <span translate=no>_^_0_^_</span> into <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3068\u306b\u5206\u5272 <span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span>.</p>\n<p>The paper notes that it's important to initialize weights to small values and the bias to <span translate=no>_^_2_^_</span>, so that during the initial training <span translate=no>_^_3_^_</span> is close to identity (apart from the split). </p>\n": "<p>\u91cd\u91cf <span translate=no>_^_0_^_</span> (\u30a4\u30f3<span translate=no>_^_1_^_</span>)</p>\n<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u91cd\u307f\u3092\u5c0f\u3055\u3044\u5024\u306b\u521d\u671f\u5316\u3057\u3001\u30d0\u30a4\u30a2\u30b9\u3092\u306b\u521d\u671f\u5316\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3067\u3042\u308b\u3068\u8ff0\u3079\u3066\u3044\u307e\u3059\u3002\u305d\u3046\u3059\u308c\u3070<span translate=no>_^_2_^_</span>\u3001\u6700\u521d\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\uff08<span translate=no>_^_3_^_</span>\u5206\u5272\u306f\u5225\u3068\u3057\u3066\uff09\u540c\u4e00\u306b\u8fd1\u3044\u3082\u306e\u306b\u306a\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span></p>\n<p>The paper notes that it's important to initialize bias to <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u91cd\u91cf (<span translate=no>_^_0_^_</span>\u30a4\u30f3) <span translate=no>_^_1_^_</span></p>\n<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30d0\u30a4\u30a2\u30b9\u3092\u306b\u521d\u671f\u5316\u3059\u308b\u3053\u3068\u304c\u91cd\u8981\u3060\u3068\u6307\u6458\u3057\u3066\u3044\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the dimensionality (<span translate=no>_^_1_^_</span>) of <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the dimensionality of <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the length of the token sequence (<span translate=no>_^_6_^_</span>)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u306e\u6b21\u5143 () <span translate=no>_^_1_^_</span> <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u306e\u6b21\u5143\u3067\u3059 <span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u306f\u30c8\u30fc\u30af\u30f3\u30fb\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055 (<span translate=no>_^_6_^_</span>)</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the dimensionality of <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the sequence length</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306e\u6b21\u5143\u3067\u3059 <span translate=no>_^_1_^_</span></li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens among each other. The last dimension of size <span translate=no>_^_5_^_</span> is the batch, which we have in other transformer implementations and was left for compatibility.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5f62\u72b6\u306e\u5165\u529b\u3067\u3059 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>is \u306f\u3001<span translate=no>_^_4_^_</span>\u30c8\u30fc\u30af\u30f3\u540c\u58eb\u306e\u53ef\u8996\u6027\u3092\u5236\u5fa1\u3059\u308b\u30d6\u30fc\u30ea\u30a2\u30f3\u30de\u30b9\u30af\u3067\u3059\u3002<span translate=no>_^_5_^_</span>\u30b5\u30a4\u30ba\u306e\u6700\u5f8c\u306e\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u306f\u30d0\u30c3\u30c1\u3067\u3059\u3002\u3053\u308c\u306f\u4ed6\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u5b9f\u88c5\u306b\u3082\u3042\u308a\u307e\u3059\u304c\u3001\u4e92\u63db\u6027\u306e\u305f\u3081\u306b\u6b8b\u3055\u308c\u3066\u3044\u307e\u3059</li></ul>\u3002\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input embedding tensor <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens among each other.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5f62\u72b6\u306e\u5165\u529b\u57cb\u3081\u8fbc\u307f\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_2_^_</span></li>\n</ul><li><span translate=no>_^_3_^_</span>\u306f\u3001<span translate=no>_^_4_^_</span>\u30c8\u30fc\u30af\u30f3\u540c\u58eb\u306e\u53ef\u8996\u6027\u3092\u5236\u5fa1\u3059\u308b\u30d6\u30fc\u30ea\u30a2\u30f3\u30b7\u30a7\u30a4\u30d7\u30de\u30b9\u30af\u3067\u3059\u3002</li>\n",
|
||||
"Pay Attention to MLPs (gMLP)": "MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044",
|
||||
"This is an annotated implementation/tutorial of Pay Attention to MLPs (gMLP) in PyTorch.": "\u3053\u308c\u306f PyTorch \u306e\u300cMLP\uff08GMLP\uff09\u306b\u6ce8\u610f\u300d\u306e\u6ce8\u91c8\u4ed8\u304d\u306e\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"<h1>Pay Attention to MLPs (gMLP)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a gMLP model based autoregressive model.</p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>MLPs(GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/2105.08050\">MLPs \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a> . </p>\n<p>\u0db8\u0dd9\u0db8\u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0db6\u0dc4\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0db4\u0dbb\u0dca\u0dc3\u0dd9\u0db4\u0dca\u0da7\u0dca\u0dbb\u0ddd\u0db1\u0dca (\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3) \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0d9a\u0dca \u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0d9f \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0d85\u0dad\u0dbb \u0d92\u0dc0\u0dcf <strong>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3</strong>\u0dbd\u0dd9\u0dc3 \u0db1\u0db8\u0dca \u0d9a\u0dbb\u0dba\u0dd2. \u0d91\u0dba <span translate=no>_^_0_^_</span> <em>GMLP</em> \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dad\u0ddc\u0d9c\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0db1\u0dca\u0dc0\u0dd2\u0dad \u0dc0\u0dda. </p>\n<p>GMLP\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf <a href=\"experiment.html\">\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dda\u0dad\u0dba</a> \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<h2>Spatial Gating Unit</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a linear transformation along the sequence dimension, and <span translate=no>_^_2_^_</span> is element-wise multiplication. <span translate=no>_^_3_^_</span> is split into to parts of equal size <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> along the channel dimension (embedding dimension).</p>\n": "<h2>\u0d85\u0dc0\u0d9a\u0dcf\u0dc1\u0dd3\u0dba\u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0d92\u0d9a\u0d9a\u0dba</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0db8\u0dcf\u0db1\u0dba \u0d94\u0dc3\u0dca\u0dc3\u0dda \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0d9a\u0dca \u0dc0\u0db1 \u0d85\u0dad\u0dbb <span translate=no>_^_2_^_</span> \u0d91\u0dba \u0db8\u0dd6\u0dbd\u0daf\u0dca\u0dbb\u0dc0\u0dca\u0dba \u0d85\u0db1\u0dd4\u0dc0 \u0d9c\u0dd4\u0dab \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2. <span translate=no>_^_1_^_</span> <span translate=no>_^_3_^_</span> \u0dc3\u0db8\u0dcf\u0db1 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dda \u0d9a\u0ddc\u0da7\u0dc3\u0dca \u0dc0\u0dbd\u0da7 <span translate=no>_^_4_^_</span> \u0dc3\u0dc4 \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0db8\u0dcf\u0db1\u0dba <span translate=no>_^_5_^_</span> \u0d94\u0dc3\u0dca\u0dc3\u0dda (\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db8\u0dcf\u0db1\u0dba) \u0db6\u0dd9\u0daf\u0dcf \u0d87\u0dad. </p>\n",
|
||||
"<h2>gMLP Block</h2>\n<p>Each block does the following transformations to input embeddings <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the sequence length and <span translate=no>_^_2_^_</span> is the dimensionality of the embeddings:</p>\n<span translate=no>_^_3_^_</span><p>where <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are learnable projection weights. <span translate=no>_^_6_^_</span> is the Spacial Gating Unit defined below. Output dimensionality of <span translate=no>_^_7_^_</span> will be half of <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span> is an activation function such as <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLU</a>.</p>\n": "<h2>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3\u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca</h2>\n<p>\u0dc3\u0dd1\u0db8\u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca \u0d91\u0d9a\u0d9a\u0dca\u0db8 \u0d86\u0daf\u0dcf\u0db1 <span translate=no>_^_0_^_</span> \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dc4\u0dad \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd9\u0db1 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0db1\u0dca \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0daf\u0dd2\u0d9c \u0dc3\u0dc4 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca <span translate=no>_^_2_^_</span> \u0dc0\u0dda: <span translate=no>_^_1_^_</span> </p>\n<span translate=no>_^_3_^_</span><p>\u0d89\u0d9c\u0dd9\u0db1 <span translate=no>_^_5_^_</span> \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2 \u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0dab \u0db6\u0dbb \u0d9a\u0ddc\u0dc4\u0dda\u0daf <span translate=no>_^_4_^_</span> \u0dc3\u0dc4 \u0d87\u0dad. <span translate=no>_^_6_^_</span> \u0db4\u0dc4\u0dad \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0d85\u0db7\u0dca\u0dba\u0dc0\u0d9a\u0dcf\u0dc1 \u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0d92\u0d9a\u0d9a\u0dba \u0dc0\u0dda. \u0db1\u0dd2\u0db8\u0dd0\u0dc0\u0dd4\u0db8\u0dca \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d85\u0da9\u0d9a\u0dca <span translate=no>_^_7_^_</span> \u0dc0\u0db1\u0dd4 \u0d87\u0dad <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span> <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLu</a>\u0dc0\u0dd0\u0db1\u0dd2 \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0d9a\u0dd2. </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span>. The batch dimension should be of size <span translate=no>_^_2_^_</span> because this implementation supports only same mask for all samples in the batch. </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba \u0d87\u0dad <span translate=no>_^_1_^_</span>. \u0db8\u0dd9\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0db8 \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0db4\u0db8\u0dab\u0d9a\u0dca \u0dc3\u0dc4\u0dcf\u0dba \u0dc0\u0db1 <span translate=no>_^_2_^_</span> \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db8\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dd9\u0db1\u0dca \u0dba\u0dd4\u0d9a\u0dca\u0dad \u0dc0\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4\u0dba. </p>\n",
|
||||
"<p>Activation function <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Add the shortcut connection </p>\n": "<p>\u0d9a\u0dd9\u0da7\u0dd2\u0db8\u0d82\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dad\u0dcf\u0dc0\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Apply mask to the weights.</p>\n<p>If <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> then <span translate=no>_^_2_^_</span> will not get any information from token <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u0db4\u0da9\u0dd2\u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0dc0\u0dbb\u0dab \u0dba\u0ddc\u0daf\u0db1\u0dca\u0db1. </p>\n<p>\u0d91\u0dc3\u0dda <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u0db1\u0db8\u0dca <span translate=no>_^_2_^_</span> \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd9\u0db1\u0dca \u0d9a\u0dd2\u0dc3\u0dd2\u0daf\u0dd4 \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0d9a\u0dca \u0dbd\u0db6\u0dcf <span translate=no>_^_3_^_</span>\u0db1\u0ddc\u0d9c\u0db1\u0dd3. </p>\n",
|
||||
"<p>Check mask </p>\n": "<p>\u0d86\u0dc0\u0dbb\u0dab\u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Embedding size (required by <a href=\"../models.html#Encoder\">Encoder</a>. We use the encoder module from transformer architecture and plug <em>gMLP</em> block as a replacement for the <a href=\"../models.html#Encoder\">Transformer Layer</a>. </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba ( <a href=\"../models.html#Encoder\">\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a>\u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dda. \u0d85\u0db4\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0dd9\u0db1\u0dca \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dbb \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"../models.html#Encoder\">\u0dc3\u0dca\u0dae\u0dbb\u0dba</a>\u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7 \u0d86\u0daf\u0dda\u0dc1\u0d9a\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 <em>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3</em> \u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca \u0db4\u0dca\u0dbd\u0d9c\u0dca \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
|
||||
"<p>Final projection <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get sequence length </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0daf\u0dd2\u0d9c \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the weight matrix; truncate if larger than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db6\u0dbb\u0d85\u0db1\u0dd4\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1; \u0dc0\u0da9\u0dcf \u0dc0\u0dd2\u0dc1\u0dcf\u0dbd \u0db1\u0db8\u0dca \u0da7\u0dca\u0dbb\u0db1\u0dca\u0d9a\u0dda\u0da7\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Here we only support the same mask for all samples </p>\n": "<p>\u0db8\u0dd9\u0db1\u0dca\u0db1\u0d85\u0db4\u0dd2 \u0dc3\u0dc4\u0dcf\u0dba \u0daf\u0dd9\u0db1\u0dca\u0db1\u0dda \u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0db8 \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0db4\u0db8\u0dab\u0dd2 </p>\n",
|
||||
"<p>Keep a copy for shortcut connection </p>\n": "<p>\u0d9a\u0dd9\u0da7\u0dd2\u0db8\u0d82\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dad\u0dcf\u0dc0\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd2\u0da7\u0db4\u0dad\u0d9a\u0dca \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Normalization layer before applying <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dba\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db4\u0dd9\u0dbb \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Normalization layer fro Pre-Norm </p>\n": "<p>\u0db4\u0dd9\u0dbb-\u0dc0\u0dd0\u0da9\u0dd9\u0db1\u0dca\u0db1\u0dda\u0dc3\u0dd2\u0da7 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u200d\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> before <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> \u0db4\u0dd9\u0dbb \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Projection and activation <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0dab\u0dba\u0dc3\u0dc4 \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Projection layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0db3\u0dc4\u0dcf\u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Remove the batch dimension </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db8\u0dcf\u0db1\u0dba \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Spacial Gating Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0db7\u0dca\u0dba\u0dc0\u0d9a\u0dcf\u0dc1\u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0d92\u0d9a\u0d9a\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Split <span translate=no>_^_0_^_</span> into <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> \u0db6\u0dd9\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> \u0dc3\u0dc4 <span translate=no>_^_2_^_</span> </p>\n",
|
||||
"<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span>.</p>\n<p>The paper notes that it's important to initialize weights to small values and the bias to <span translate=no>_^_2_^_</span>, so that during the initial training <span translate=no>_^_3_^_</span> is close to identity (apart from the split). </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc3\u0dd2\u0dbb\u0dd4\u0dbb\u0dda \u0db6\u0dbb <span translate=no>_^_1_^_</span>. </p>\n<p>\u0d9a\u0dd4\u0da9\u0dcf\u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc4\u0dcf \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0db6\u0dbb \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc0\u0dd0\u0daf\u0d9c\u0dad\u0dca \u0db6\u0dc0 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0d9a\u0dbb\u0dba\u0dd2 <span translate=no>_^_2_^_</span>, \u0d91\u0dc0\u0dd2\u0da7 \u0db8\u0dd6\u0dbd\u0dd2\u0d9a \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dad\u0dd4\u0dc5 \u0d85\u0db1\u0db1\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba\u0da7 \u0dc3\u0db8\u0dd3\u0db4 <span translate=no>_^_3_^_</span> \u0dc0\u0dda (\u0db7\u0dda\u0daf\u0dba \u0dc4\u0dd0\u0dbb\u0dd4\u0dab\u0dd4 \u0dc0\u0dd2\u0da7). </p>\n",
|
||||
"<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span></p>\n<p>The paper notes that it's important to initialize bias to <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u0dc3\u0dd2\u0dbb\u0dd4\u0dbb\u0dda\u0db6\u0dbb <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n<p>\u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3\u0dc0\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc0\u0dd0\u0daf\u0d9c\u0dad\u0dca \u0db6\u0dc0 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc3\u0da7\u0dc4\u0db1\u0dca <span translate=no>_^_2_^_</span>\u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the dimensionality (<span translate=no>_^_1_^_</span>) of <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the dimensionality of <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the length of the token sequence (<span translate=no>_^_6_^_</span>)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca (<span translate=no>_^_1_^_</span>) <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> \u0dc4\u0dd2 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc0\u0dda <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda \u0daf\u0dd2\u0d9c (<span translate=no>_^_6_^_</span>)</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the dimensionality of <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the sequence length</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd2 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc0\u0dda <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba \u0daf\u0dd2\u0d9c \u0dc0\u0dda</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens among each other. The last dimension of size <span translate=no>_^_5_^_</span> is the batch, which we have in other transformer implementations and was left for compatibility.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda <span translate=no>_^_1_^_</span> \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dc0\u0dda <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> \u0dba\u0db1\u0dd4 \u0d91\u0d9a\u0dd2\u0db1\u0dd9\u0d9a\u0dcf \u0d85\u0dad\u0dbb \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dbd \u0daf\u0dd8\u0dc1\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db4\u0dcf\u0dbd\u0db1\u0dba <span translate=no>_^_4_^_</span> \u0d9a\u0dbb\u0db1 \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0db6\u0dd6\u0dbd\u0dd2\u0dba\u0db1\u0dca \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dd2. \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dda \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db8\u0dcf\u0db1\u0dba <span translate=no>_^_5_^_</span> \u0dc0\u0db1\u0dca\u0db1\u0dda \u0dc0\u0dd9\u0db1\u0dad\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0dc0\u0dbd\u0daf\u0dd3 \u0d85\u0db4 \u0dc3\u0dad\u0dd4\u0dc0 \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0d85\u0db1\u0dd4\u0d9a\u0dd6\u0dbd\u0dad\u0dcf\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0dc0 \u0d87\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dba\u0dd2. </li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input embedding tensor <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens among each other.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d86\u0daf\u0dcf\u0db1 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 <span translate=no>_^_1_^_</span> \u0dc0\u0dda <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> \u0dba\u0db1\u0dd4 \u0d91\u0d9a\u0dd2\u0db1\u0dd9\u0d9a\u0dcf \u0d85\u0dad\u0dbb \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dbd \u0daf\u0dd8\u0dc1\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db4\u0dcf\u0dbd\u0db1\u0dba <span translate=no>_^_4_^_</span> \u0d9a\u0dbb\u0db1 \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0db6\u0dd6\u0dbd\u0dd2\u0dba\u0db1\u0dca \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dd2. </li></ul>\n",
|
||||
"Pay Attention to MLPs (gMLP)": "MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1",
|
||||
"This is an annotated implementation/tutorial of Pay Attention to MLPs (gMLP) in PyTorch.": "\u0db8\u0dd9\u0dba PyTorch \u0dc4\u0dd2 MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc0\u0dd2\u0db1\u0dba\u0d9a\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba\u0d9a\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"<h1>Pay Attention to MLPs (gMLP)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a gMLP model based autoregressive model.</p>\n": "<h1>\u6ce8\u610f MLP (GmLP)</h1>\n<p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u5bf9\u300a<a href=\"https://arxiv.org/abs/2105.08050\">\u6ce8\u610f MLP\u300b\u4e00\u6587\u7684</a>\u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u4ecb\u7ecd\u4e86\u4e00\u79cd\u57fa\u4e8e\u591a\u5c42\u611f\u77e5\u5668\uff08MLP\uff09\u7684\u5e26\u6709\u95e8\u63a7\u7684\u67b6\u6784\uff0c\u4ed6\u4eec\u5c06\u5176\u547d\u540d\u4e3a <strong>gmLP</strong>\u3002\u5b83\u7531\u4e00\u5806<span translate=no>_^_0_^_</span> <em>gmLP</em> \u5757\u7ec4\u6210\u3002</p>\n<p>\u8fd9\u662f\u57fa<a href=\"experiment.html\">\u4e8e GmLP \u6a21\u578b\u7684\u81ea\u56de\u5f52\u6a21\u578b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
|
||||
"<h2>Spatial Gating Unit</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a linear transformation along the sequence dimension, and <span translate=no>_^_2_^_</span> is element-wise multiplication. <span translate=no>_^_3_^_</span> is split into to parts of equal size <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> along the channel dimension (embedding dimension).</p>\n": "<h2>\u7a7a\u95f4\u95e8\u63a7\u5355\u5143</h2>\n<p><span translate=no>_^_0_^_</span></p>\n<p>\u5176\u4e2d\uff0c<span translate=no>_^_1_^_</span>\u662f\u6cbf\u5e8f\u5217\u7ef4\u5ea6\u7684\u7ebf\u6027\u53d8\u6362\uff0c<span translate=no>_^_2_^_</span>\u662f\u9010\u5143\u7d20\u4e58\u6cd5\u3002<span translate=no>_^_3_^_</span>\u88ab\u5206\u6210\u4e24\u4e2a\u5927\u5c0f\u76f8\u7b49\u7684\u90e8\u5206\uff0c<span translate=no>_^_4_^_</span>\u5e76<span translate=no>_^_5_^_</span>\u6cbf\u7740\u901a\u9053\u5c3a\u5bf8\uff08\u5d4c\u5165\u7ef4\u5ea6\uff09\u3002</p>\n",
|
||||
"<h2>gMLP Block</h2>\n<p>Each block does the following transformations to input embeddings <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the sequence length and <span translate=no>_^_2_^_</span> is the dimensionality of the embeddings:</p>\n<span translate=no>_^_3_^_</span><p>where <span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are learnable projection weights. <span translate=no>_^_6_^_</span> is the Spacial Gating Unit defined below. Output dimensionality of <span translate=no>_^_7_^_</span> will be half of <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span> is an activation function such as <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLU</a>.</p>\n": "<h2>gmLP Block</h2>\n<p>\u6bcf\u4e2a\u6a21\u5757\u5bf9\u8f93\u5165\u5d4c\u5165\u8fdb\u884c\u4ee5\u4e0b\u8f6c\u6362\uff0c<span translate=no>_^_0_^_</span>\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u5e8f\u5217\u957f\u5ea6\uff0c<span translate=no>_^_2_^_</span>\u662f\u5d4c\u5165\u7684\u7ef4\u5ea6\uff1a</p>\n<span translate=no>_^_3_^_</span><p>\u5176\u4e2d<span translate=no>_^_4_^_</span>\u548c<span translate=no>_^_5_^_</span>\u662f\u53ef\u5b66\u4e60\u7684\u6295\u5f71\u6743\u91cd\u3002<span translate=no>_^_6_^_</span>\u662f\u4e0b\u9762\u5b9a\u4e49\u7684\u7a7a\u95f4\u95e8\u63a7\u5355\u5143\u3002\u7684\u8f93\u51fa\u7ef4\u5ea6<span translate=no>_^_7_^_</span>\u5c06\u4e3a\u7684\u4e00\u534a<span translate=no>_^_8_^_</span>\u3002<span translate=no>_^_9_^_</span>\u662f\u4e00\u4e2a\u6fc0\u6d3b\u51fd\u6570\uff0c\u6bd4\u5982 <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.GELU.html\">GeLU</a>\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span>. The batch dimension should be of size <span translate=no>_^_2_^_</span> because this implementation supports only same mask for all samples in the batch. </p>\n": "<p><span translate=no>_^_0_^_</span>\u6709\u5f62\u72b6<span translate=no>_^_1_^_</span>\u3002\u6279\u6b21\u7ef4\u5ea6\u5e94\u4e3a size\uff0c<span translate=no>_^_2_^_</span>\u56e0\u4e3a\u6b64\u5b9e\u73b0\u4ec5\u652f\u6301\u6279\u6b21\u4e2d\u6240\u6709\u6837\u672c\u7684\u76f8\u540c\u63a9\u7801\u3002</p>\n",
|
||||
"<p>Activation function <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6fc0\u6d3b\u529f\u80fd<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Add the shortcut connection </p>\n": "<p>\u6dfb\u52a0\u5feb\u6377\u65b9\u5f0f\u8fde\u63a5</p>\n",
|
||||
"<p>Apply mask to the weights.</p>\n<p>If <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> then <span translate=no>_^_2_^_</span> will not get any information from token <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u5c06\u906e\u7f69\u5e94\u7528\u4e8e\u6743\u91cd\u3002</p>\n<p>\u5982\u679c<span translate=no>_^_0_^_</span>\u662f<span translate=no>_^_1_^_</span>\uff0c\u5219<span translate=no>_^_2_^_</span>\u4e0d\u4f1a\u4ece\u4ee4\u724c\u4e2d\u83b7\u53d6\u4efb\u4f55\u4fe1\u606f<span translate=no>_^_3_^_</span>\u3002</p>\n",
|
||||
"<p>Check mask </p>\n": "<p>\u68c0\u67e5\u53e3\u7f69</p>\n",
|
||||
"<p>Embedding size (required by <a href=\"../models.html#Encoder\">Encoder</a>. We use the encoder module from transformer architecture and plug <em>gMLP</em> block as a replacement for the <a href=\"../models.html#Encoder\">Transformer Layer</a>. </p>\n": "<p>\u5d4c\u5165\u5927\u5c0f\uff08\u7f16<a href=\"../models.html#Encoder\">\u7801\u5668</a>\u9700\u8981\u3002\u6211\u4eec\u4f7f\u7528\u53d8\u538b\u5668\u67b6\u6784\u4e2d\u7684\u7f16\u7801\u5668\u6a21\u5757\uff0c\u5e76\u63d2\u5165 <em>GmLP</em> \u6a21\u5757\u4f5c\u4e3a<a href=\"../models.html#Encoder\">\u53d8\u538b\u5668\u5c42</a>\u7684\u66ff\u4ee3\u54c1\u3002</p>\n",
|
||||
"<p>Final projection <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u7ec8\u6295\u5f71<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get sequence length </p>\n": "<p>\u83b7\u53d6\u5e8f\u5217\u957f\u5ea6</p>\n",
|
||||
"<p>Get the weight matrix; truncate if larger than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u83b7\u53d6\u6743\u91cd\u77e9\u9635\uff1b\u5982\u679c\u5927\u4e8e<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Here we only support the same mask for all samples </p>\n": "<p>\u8fd9\u91cc\u6211\u4eec\u53ea\u652f\u6301\u6240\u6709\u6837\u672c\u4f7f\u7528\u76f8\u540c\u7684\u63a9\u7801</p>\n",
|
||||
"<p>Keep a copy for shortcut connection </p>\n": "<p>\u4fdd\u7559\u4e00\u4efd\u7528\u4e8e\u5feb\u6377\u65b9\u5f0f\u8fde\u63a5\u7684\u526f\u672c</p>\n",
|
||||
"<p>Normalization layer before applying <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e94\u7528\u4e4b\u524d\u7684\u6807\u51c6\u5316\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Normalization layer fro Pre-Norm </p>\n": "<p>Pre-Norm \u7684\u6807\u51c6\u5316\u5c42</p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u89c4\u8303\u5316<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> before <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u4e4b\u524d\u8fdb\u884c\u6807\u51c6\u5316<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Projection and activation <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6295\u5c04\u548c\u6fc0\u6d3b<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Projection layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6295\u5f71\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Remove the batch dimension </p>\n": "<p>\u79fb\u9664\u6279\u91cf\u7ef4\u5ea6</p>\n",
|
||||
"<p>Spacial Gating Unit <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7a7a\u95f4\u95e8\u63a7\u5355\u5143<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Split <span translate=no>_^_0_^_</span> into <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> </p>\n": "<p>\u62c6<span translate=no>_^_0_^_</span>\u5206\u4e3a<span translate=no>_^_1_^_</span>\u548c<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span>.</p>\n<p>The paper notes that it's important to initialize weights to small values and the bias to <span translate=no>_^_2_^_</span>, so that during the initial training <span translate=no>_^_3_^_</span> is close to identity (apart from the split). </p>\n": "<p>\u91cd\u91cf<span translate=no>_^_0_^_</span>\u5728<span translate=no>_^_1_^_</span>\u3002</p>\n<p>\u672c\u6587\u6307\u51fa\uff0c\u91cd\u8981\u7684\u662f\u5c06\u6743\u91cd\u521d\u59cb\u5316\u4e3a\u8f83\u5c0f\u7684\u503c\uff0c\u5e76\u5c06\u504f\u5dee\u521d\u59cb\u5316<span translate=no>_^_3_^_</span>\u4e3a<span translate=no>_^_2_^_</span>\uff0c\u8fd9\u6837\u5728\u521d\u59cb\u8bad\u7ec3\u671f\u95f4\u5c31\u63a5\u8fd1\u8eab\u4efd\uff08\u62c6\u5206\u9664\u5916\uff09\u3002</p>\n",
|
||||
"<p>Weight <span translate=no>_^_0_^_</span> in <span translate=no>_^_1_^_</span></p>\n<p>The paper notes that it's important to initialize bias to <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u91cd\u91cf<span translate=no>_^_0_^_</span>\u5728<span translate=no>_^_1_^_</span></p>\n<p>\u672c\u6587\u6307\u51fa\uff0c\u5c06\u504f\u89c1\u521d\u59cb\u5316\u4e3a<span translate=no>_^_2_^_</span>.</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the dimensionality (<span translate=no>_^_1_^_</span>) of <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the dimensionality of <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the length of the token sequence (<span translate=no>_^_6_^_</span>)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u7684\u7ef4\u5ea6 (<span translate=no>_^_1_^_</span>)<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f\u7684\u7ef4\u5ea6<span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u662f\u4ee4\u724c\u5e8f\u5217\u7684\u957f\u5ea6 (<span translate=no>_^_6_^_</span>)</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the dimensionality of <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the sequence length</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u7684\u7ef4\u5ea6<span translate=no>_^_1_^_</span></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u5e8f\u5217\u957f\u5ea6</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens among each other. The last dimension of size <span translate=no>_^_5_^_</span> is the batch, which we have in other transformer implementations and was left for compatibility.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6<span translate=no>_^_1_^_</span>\u7684\u8f93\u5165<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>is \u662f\u5f62\u72b6\u7684\u5e03\u5c14\u63a9\u7801<span translate=no>_^_4_^_</span>\uff0c\u7528\u4e8e\u63a7\u5236\u6807\u8bb0\u5728\u5f7c\u6b64\u4e4b\u95f4\u7684\u53ef\u89c1\u6027\u3002\u5c3a\u5bf8\u7684\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6<span translate=no>_^_5_^_</span>\u662f\u6279\u6b21\uff0c\u8fd9\u662f\u6211\u4eec\u5728\u5176\u4ed6\u53d8\u538b\u5668\u5b9e\u73b0\u4e2d\u4f7f\u7528\u7684\uff0c\u4e3a\u4e86\u517c\u5bb9\u6027\u800c\u7559\u4e0b\u3002</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input embedding tensor <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a boolean mask of shape <span translate=no>_^_4_^_</span> that controls the visibility of tokens among each other.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u7684\u8f93\u5165\u5d4c\u5165<span translate=no>_^_1_^_</span>\u5f20\u91cf<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f\u5f62\u72b6\u7684\u5e03\u5c14\u63a9\u7801<span translate=no>_^_4_^_</span>\uff0c\u7528\u4e8e\u63a7\u5236\u6807\u8bb0\u5728\u5f7c\u6b64\u4e4b\u95f4\u7684\u53ef\u89c1\u6027\u3002</li></ul>\n",
|
||||
"Pay Attention to MLPs (gMLP)": "\u6ce8\u610f MLP (gMLP)",
|
||||
"This is an annotated implementation/tutorial of Pay Attention to MLPs (gMLP) in PyTorch.": "\u8fd9\u662f PyTorch \u4e2d\u6ce8\u610f MLP\uff08GmLP\uff09\u7684\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0/\u6559\u7a0b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Pay Attention to MLPs (gMLP)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">gMLP model</a>. The paper also applies a Stochastic Depth regularization where some layers are removed randomly during training. We have not implemented that here.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h1><a href=\"index.html\">MLP (gMLP) \u306e\u5b9f\u9a13\u306b\u3054\u6ce8\u76ee\u304f\u3060\u3055\u3044</a></h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f gMLP \u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30a2\u30ce\u30c6\u30fc\u30b7\u30e7\u30f3\u4ed8\u304d PyTorch \u5b9f\u9a13\u3067\u3059\u3002</a>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\u4e00\u90e8\u306e\u5c64\u304c\u30e9\u30f3\u30c0\u30e0\u306b\u524a\u9664\u3055\u308c\u308b\u78ba\u7387\u7684\u6df1\u5ea6\u6b63\u5247\u5316\u3082\u9069\u7528\u3057\u3066\u3044\u307e\u3059\u3002\u3053\u3053\u3067\u306f\u5b9f\u88c5\u3057\u3066\u3044\u307e\u305b\u3093\u3002</p>\n<p>\u3053\u308c\u306f\u3001<a href=\"../basic/autoregressive_experiment.html\">\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u5df1\u56de\u5e30NLP\u30bf\u30b9\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u69cb\u6210\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u3001<a href=\"../basic/autoregressive_transformer.html\">\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u5df1\u56de\u5e30NLP\u30bf\u30b9\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u69cb\u6210\u304b\u3089\u7d99\u627f\u3055\u308c\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h3>Create a gMLP block</h3>\n": "<h3>gMLP \u30d6\u30ed\u30c3\u30af\u3092\u4f5c\u6210\u3059\u308b</h3>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> for gMLP projection layer </p>\n": "<p><span translate=no>_^_0_^_</span>gMLP \u30d7\u30ed\u30b8\u30a7\u30af\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u7528</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Replace the encoder layer with a gMLP layer </p>\n": "<p>\u30a8\u30f3\u30b3\u30fc\u30c0\u5c64\u3092gMLP\u5c64\u306b\u7f6e\u304d\u63db\u3048\u308b</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<p>gMLP Block </p>\n": "<p>GmLP \u30d6\u30ed\u30c3\u30af</p>\n",
|
||||
"Pay Attention to MLPs (gMLP) Experiment": "MLP (gMLP) \u306e\u5b9f\u9a13\u306b\u3054\u6ce8\u76ee\u304f\u3060\u3055\u3044",
|
||||
"This experiment trains a gMLP based model on Tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u57fa\u306b gMLP \u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Pay Attention to MLPs (gMLP)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">gMLP model</a>. The paper also applies a Stochastic Depth regularization where some layers are removed randomly during training. We have not implemented that here.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">MLPs (GMLP) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4</a> \u0d9a\u0dbb\u0db1\u0dca\u0db1</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">GMLP \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0dbb\u0dca\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0dc3\u0db8\u0dc4\u0dbb \u0dc3\u0dca\u0dae\u0dbb \u0d85\u0dc4\u0db9\u0dd4 \u0dbd\u0dd9\u0dc3 \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0db6\u0db1 Stochastic Depth regularization \u0daf \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d85\u0daf\u0dcf\u0dc5 \u0dc0\u0dda. \u0d85\u0db4\u0dd2 \u0d91\u0dba \u0db8\u0dd9\u0dc4\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb \u0db1\u0dd0\u0dad. </p>\n<p>\u0db8\u0dd9\u0dba\u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda <a href=\"../basic/autoregressive_experiment.html\">\u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba-\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0db8\u0dad \u0dba. </p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba <a href=\"../basic/autoregressive_transformer.html\">\u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba-\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9a\u0dbb\u0dca\u0dad\u0dc0\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0dda. </p>\n",
|
||||
"<h3>Create a gMLP block</h3>\n": "<h3>GMLP\u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca \u0d91\u0d9a\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> for gMLP projection layer </p>\n": "<p><span translate=no>_^_0_^_</span> GMLP \u0db4\u0dca\u0dbb\u0d9a\u0dca\u0dc2\u0dda\u0db4\u0dab \u0dc3\u0dca\u0dae\u0dbb\u0dba \u0dc3\u0db3\u0dc4\u0dcf </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Replace the encoder layer with a gMLP layer </p>\n": "<p>\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca\u0dc3\u0dca\u0dad\u0dbb\u0dba GMLP \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set model size </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"<p>gMLP Block </p>\n": "<p>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3\u0db6\u0dca\u0dbd\u0ddc\u0d9a\u0dca </p>\n",
|
||||
"Pay Attention to MLPs (gMLP) Experiment": "MLPs (GMLP) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1",
|
||||
"This experiment trains a gMLP based model on Tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db8\u0dad GMLP \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Pay Attention to MLPs (gMLP)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">gMLP model</a>. The paper also applies a Stochastic Depth regularization where some layers are removed randomly during training. We have not implemented that here.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h1><a href=\"index.html\">\u6ce8\u610f mLP (gmLP)</a> \u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3 <a href=\"index.html\">gmLP \u6a21\u578b</a>\u3002\u672c\u6587\u8fd8\u5e94\u7528\u4e86\u968f\u673a\u6df1\u5ea6\u6b63\u5219\u5316\uff0c\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u4f1a\u968f\u673a\u5220\u9664\u4e00\u4e9b\u56fe\u5c42\u3002\u6211\u4eec\u6ca1\u6709\u5728\u8fd9\u91cc\u5b9e\u73b0\u8fd9\u4e00\u70b9\u3002</p>\n<p>\u8fd9\u662f\u57fa\u4e8e<a href=\"../basic/autoregressive_experiment.html\">\u7b80\u5355\u53d8\u6362\u5668\u81ea\u56de\u5f52 NLP \u4efb\u52a1\u7684\u8bad\u7ec3\u5faa\u73af\u548c\u914d\u7f6e</a>\u3002</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea<a href=\"../basic/autoregressive_transformer.html\">\u8bad\u7ec3\u5faa\u73af\u548c\u7b80\u5355\u53d8\u538b\u5668\u81ea\u56de\u5f52\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4efb\u52a1\u7684\u914d\u7f6e</a>\u3002</p>\n",
|
||||
"<h3>Create a gMLP block</h3>\n": "<h3>\u521b\u5efa\u4e00\u4e2a GmLP \u533a\u5757</h3>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> for gMLP projection layer </p>\n": "<p><span translate=no>_^_0_^_</span>\u7528\u4e8e gMLP \u6295\u5f71\u5c42</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u578b\u53f7\u5c3a\u5bf8</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Replace the encoder layer with a gMLP layer </p>\n": "<p>\u5c06\u7f16\u7801\u5668\u5c42\u66ff\u6362\u4e3a GmLP \u5c42</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set model size </p>\n": "<p>\u8bbe\u7f6e\u6a21\u578b\u5c3a\u5bf8</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"<p>gMLP Block </p>\n": "<p>gmLP Block</p>\n",
|
||||
"Pay Attention to MLPs (gMLP) Experiment": "\u6ce8\u610f MLP (gMLP) \u5b9e\u9a8c",
|
||||
"This experiment trains a gMLP based model on Tiny Shakespeare dataset.": "\u672c\u5b9e\u9a8c\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u57fa\u4e8e GmLP \u7684\u6a21\u578b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">the training code</a> for a gMLP model based autoregressive model. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044</a></h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/2105.08050\">MLP\u306b\u6ce8\u610f\u3057\u3066</a>\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a>\u3002</p>\n<p><strong>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30b2\u30fc\u30c6\u30a3\u30f3\u30b0\u3092\u5099\u3048\u305f\u591a\u5c64\u30d1\u30fc\u30bb\u30d7\u30c8\u30ed\u30f3\uff08MLP\uff09\u30d9\u30fc\u30b9\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\uff08GMLP\u3068\u540d\u4ed8\u3051\u3089\u308c\u3066\u3044\u307e\u3059\uff09\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002</strong><span translate=no>_^_0_^_</span><em>gMLP</em> \u30d6\u30ed\u30c3\u30af\u306e\u30b9\u30bf\u30c3\u30af\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059</p>\u3002\n<p><a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">gMLP\u30e2\u30c7\u30eb\u30d9\u30fc\u30b9\u306e\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059</a>\u3002</p>\n",
|
||||
"Pay Attention to MLPs (gMLP)": "MLP (GMLP) \u306b\u3054\u6ce8\u610f\u304f\u3060\u3055\u3044"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">the training code</a> for a gMLP model based autoregressive model.</p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_1_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a></h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/2105.08050\">MLPs \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</a> . </p>\n<p>\u0db8\u0dd9\u0db8\u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0db6\u0dc4\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0db4\u0dbb\u0dca\u0dc3\u0dd9\u0db4\u0dca\u0da7\u0dca\u0dbb\u0ddd\u0db1\u0dca (\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3) \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0d9a\u0dca \u0d9c\u0dda\u0da7\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0d9f \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0d85\u0dad\u0dbb \u0d92\u0dc0\u0dcf <strong>\u0da2\u0dd3\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3</strong>\u0dbd\u0dd9\u0dc3 \u0db1\u0db8\u0dca \u0d9a\u0dbb\u0dba\u0dd2. \u0d91\u0dba <span translate=no>_^_0_^_</span> <em>GMLP</em> \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dad\u0ddc\u0d9c\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0db1\u0dca\u0dc0\u0dd2\u0dad \u0dc0\u0dda. </p>\n<p>GMLP\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf <a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dda\u0dad\u0dba</a> \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n<p><a href=\"https://app.labml.ai/run/01bd941ac74c11eb890c1d9196651a4a\"><span translate=no>_^_1_^_</span></a> </p>\n",
|
||||
"Pay Attention to MLPs (gMLP)": "MLPs (GMLP) \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">Pay Attention to MLPs (gMLP)</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.08050\">Pay Attention to MLPs</a>.</p>\n<p>This paper introduces a Multilayer Perceptron (MLP) based architecture with gating, which they name <strong>gMLP</strong>. It consists of a stack of <span translate=no>_^_0_^_</span> <em>gMLP</em> blocks.</p>\n<p>Here is <a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">the training code</a> for a gMLP model based autoregressive model. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/gmlp/index.html\">\u6ce8\u610f MLP (GmLP)</a></h1>\n<p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u5bf9\u300a<a href=\"https://arxiv.org/abs/2105.08050\">\u6ce8\u610f MLP\u300b\u4e00\u6587\u7684</a>\u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u4ecb\u7ecd\u4e86\u4e00\u79cd\u57fa\u4e8e\u591a\u5c42\u611f\u77e5\u5668\uff08MLP\uff09\u7684\u5e26\u6709\u95e8\u63a7\u7684\u67b6\u6784\uff0c\u4ed6\u4eec\u5c06\u5176\u547d\u540d\u4e3a <strong>gmLP</strong>\u3002\u5b83\u7531\u4e00\u5806<span translate=no>_^_0_^_</span> <em>gmLP</em> \u5757\u7ec4\u6210\u3002</p>\n<p>\u8fd9\u662f\u57fa<a href=\"https://nn.labml.ai/transformers/gmlp/experiment.html\">\u4e8e GmLP \u6a21\u578b\u7684\u81ea\u56de\u5f52\u6a21\u578b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
|
||||
"Pay Attention to MLPs (gMLP)": "\u6ce8\u610f MLP (gMLP)"
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"<h1>GPT</h1>\n<p>This is a tutorial/implementation of <a href=\"https://openai.com/blog/better-language-models/\">OpenAI GPT architecture</a> in <a href=\"https://pytorch.org\">PyTorch</a>. We got a bunch of implementation details from <a href=\"https://github.com/karpathy/minGPT\">minGPT</a> by <a href=\"https://twitter.com/karpathy\">@karpathy</a>. This implementation also uses character tiny shakespeare dataset.</p>\n<p>GPT model is essentially a standard transformer with a few tweaks. GPT-2 and especially GPT-3 models are quite large and won't fit on a single GPU and will need model parallelism. This implementation doesn't even use data parallelism and is intended to be more of a tutorial.</p>\n<p>Main differences of this compared to a simple autoregressive transformer are the parameter initialization, weight decay, and learning rate schedule. For the transformer we reuse the <a href=\"../transformers/index.html\">existing labml/nn transformer implementation</a>.</p>\n<p>Here's a notebook for training a GPT model on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/gpt/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>GPT</h1>\n</a><p><a href=\"https://pytorch.org\">\u3053\u308c\u306f PyTorch \u306b\u304a\u3051\u308b <a href=\"https://openai.com/blog/better-language-models/\">OpenAI GPT \u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb/\u5b9f\u88c5\u3067\u3059\u3002</a><a href=\"https://twitter.com/karpathy\">@karpathy \u306b\u3088\u3063\u3066 <a href=\"https://github.com/karpathy/minGPT\">MingPT</a> \u304b\u3089\u5b9f\u88c5\u306e\u8a73\u7d30\u3092\u305f\u304f\u3055\u3093\u5f97\u307e\u3057\u305f\u3002</a>\u3053\u306e\u5b9f\u88c5\u3067\u306f\u3001\u6587\u5b57\u30b5\u30a4\u30ba\u306e\u5c0f\u3055\u3044\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3082\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</p>\u3002\n<p>GPT\u30e2\u30c7\u30eb\u306f\u57fa\u672c\u7684\u306b\u3001\u3044\u304f\u3064\u304b\u306e\u8abf\u6574\u3092\u52a0\u3048\u305f\u6a19\u6e96\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3067\u3059\u3002GPT-2\u3001\u7279\u306bGPT-3\u306e\u30e2\u30c7\u30eb\u306f\u975e\u5e38\u306b\u5927\u304d\u304f\u3001\u5358\u4e00\u306eGPU\u306b\u306f\u53ce\u307e\u3089\u306a\u3044\u305f\u3081\u3001\u30e2\u30c7\u30eb\u306e\u4e26\u5217\u51e6\u7406\u304c\u5fc5\u8981\u306b\u306a\u308a\u307e\u3059\u3002\u3053\u306e\u5b9f\u88c5\u306f\u30c7\u30fc\u30bf\u4e26\u5217\u51e6\u7406\u3059\u3089\u4f7f\u7528\u305b\u305a\u3001\u3069\u3061\u3089\u304b\u3068\u3044\u3046\u3068\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u306e\u3088\u3046\u306a\u3082\u306e\u3067\u3059</p>\u3002\n<p>\u5358\u7d14\u306a\u81ea\u5df1\u56de\u5e30\u5909\u63db\u5668\u3068\u306e\u4e3b\u306a\u9055\u3044\u306f\u3001\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u521d\u671f\u5316\u3001\u91cd\u307f\u306e\u6e1b\u8870\u3001\u5b66\u7fd2\u7387\u306e\u30b9\u30b1\u30b8\u30e5\u30fc\u30eb\u3067\u3059\u3002\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306b\u306f\u3001<a href=\"../transformers/index.html\">\u65e2\u5b58\u306elabml/nn\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u518d\u5229\u7528\u3057\u307e\u3059</a></p>\u3002\n<p>\u3053\u308c\u306f\u3001Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067GPT\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30ce\u30fc\u30c8\u30d6\u30c3\u30af\u3067\u3059\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/gpt/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u4ee5\u4e0b\u304b\u3089\u7d99\u627f\u3055\u308c\u307e\u3059 <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>GPT model</h2>\n<p>This consists of a token embedding layer, transformer encoder, and a final linear layer that gives token logits.</p>\n": "<h2>GPT \u30e2\u30c7\u30eb</h2>\n<p>\u3053\u308c\u306f\u3001\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u5c64\u3001\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u3001\u304a\u3088\u3073\u30c8\u30fc\u30af\u30f3\u30ed\u30b8\u30c3\u30c8\u3092\u63d0\u4f9b\u3059\u308b\u6700\u5f8c\u306e\u7dda\u5f62\u5c64\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Create custom optimizer with weight decay</h3>\n<p>This code is taken from <a href=\"https://github.com/karpathy/minGPT\">minGPT</a>. This applies weight decay only to weights of linear layers.</p>\n": "<h3>\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u3092\u542b\u3080\u30ab\u30b9\u30bf\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u4f5c\u6210</h3>\n<p><a href=\"https://github.com/karpathy/minGPT\">\u3053\u306e\u30b3\u30fc\u30c9\u306fMingPT\u304b\u3089\u53d6\u5f97\u3057\u305f\u3082\u306e\u3067\u3059</a>\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u306f\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc\u306e\u30a6\u30a7\u30a4\u30c8\u306b\u306e\u307f\u9069\u7528\u3055\u308c\u307e\u3059</p>\u3002\n",
|
||||
"<h3>Initialize weights</h3>\n<p>Weights of linear layers and embedding layers are initialized to <span translate=no>_^_0_^_</span> instead of the default Xavier initialzation.</p>\n": "<h3>\u30a6\u30a7\u30a4\u30c8\u3092\u521d\u671f\u5316</h3>\n<p>\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc\u3068\u57cb\u3081\u8fbc\u307f\u30ec\u30a4\u30e4\u30fc\u306e\u91cd\u307f\u306f\u3001\u30c7\u30d5\u30a9\u30eb\u30c8\u306e Xavier <span translate=no>_^_0_^_</span> \u521d\u671f\u5316\u306e\u4ee3\u308f\u308a\u306b\u521d\u671f\u5316\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create GPT model and initialize weights</p>\n": "<p>GPT \u30e2\u30c7\u30eb\u306e\u4f5c\u6210\u3068\u91cd\u307f\u306e\u521d\u671f\u5316</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Apply custom weight initialization </p>\n": "<p>\u30ab\u30b9\u30bf\u30e0\u30a6\u30a7\u30a4\u30c8\u521d\u671f\u5316\u3092\u9069\u7528</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Collect names of parameters to apply weight decay </p>\n": "<p>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u540d\u524d\u3092\u53ce\u96c6\u3057\u3066\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u3092\u9069\u7528\u3059\u308b</p>\n",
|
||||
"<p>Create a <a href=\"../optimizers/configs.html#OptimizerConfigs\">configurable optimizer</a>, so that we can change these simply by passing a config dictionary. </p>\n": "<p><a href=\"../optimizers/configs.html#OptimizerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f5c\u6210\u3057\u3066</a>\u3001\u8a2d\u5b9a\u8f9e\u66f8\u3092\u6e21\u3059\u3060\u3051\u3067\u3053\u308c\u3089\u3092\u5909\u66f4\u3067\u304d\u308b\u3088\u3046\u306b\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u30de\u30b9\u30af\u304c\u521d\u671f\u5316\u3055\u308c\u3066\u3044\u306a\u3044\u5834\u5408\u3084\u30de\u30b9\u30af\u306e\u30b5\u30a4\u30ba\u304c\u7570\u306a\u308b\u5834\u5408\u306f\u3001\u5f8c\u7d9a\u306e\u30de\u30b9\u30af\u3092\u4f5c\u6210\u3057\u307e\u3059</p>\n",
|
||||
"<p>Custom optimizer </p>\n": "<p>\u30ab\u30b9\u30bf\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT \u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT \u306f GELU \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u3092\u4f7f\u7528\u3057\u3066\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u3092\u884c\u3044\u307e\u3059</p>\n",
|
||||
"<p>GPT uses a maximum learning rate of <span translate=no>_^_0_^_</span>. </p>\n": "<p>GPT \u306e\u6700\u5927\u5b66\u7fd2\u7387\u306f\u3067\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get all the parameters </p>\n": "<p>\u3059\u3079\u3066\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Initialize biases to <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30a4\u30a2\u30b9\u3092\u521d\u671f\u5316 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of tokens for wamup </p>\n": "<p>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u7528\u306e\u30c8\u30fc\u30af\u30f3\u6570</p>\n",
|
||||
"<p>Number of warmup optimization steps </p>\n": "<p>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6700\u9069\u5316\u30b9\u30c6\u30c3\u30d7\u306e\u6570</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Parameters that are not decayed </p>\n": "<p>\u6e1b\u8870\u3057\u306a\u3044\u30d1\u30e9\u30e1\u30fc\u30bf</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3057\u307e\u3059\uff08\u30c8\u30ec\u30fc\u30ca\u30fc\u306fRNN\u3067\u3082\u4f7f\u7528\u3055\u308c\u308b\u305f\u3081\u30012\u756a\u76ee\u306e\u5024\u306f\u72b6\u614b\u7528\u3067\u3059\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set default weight decay. This is not required since we set the weight decay in the parameter groups. </p>\n": "<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u3067\u30a6\u30a7\u30a4\u30c8\u30c7\u30a3\u30b1\u30a4\u3092\u8a2d\u5b9a\u3057\u3066\u3044\u308b\u306e\u3067\u3001\u3053\u308c\u306f\u5fc5\u9808\u3067\u306f\u3042\u308a\u307e\u305b\u3093</p>\u3002\n",
|
||||
"<p>Set model embedding size, required if we use <a href=\"../optimizers/noam.html\">Noam optimizer</a> which has an exponential decay. </p>\n": "<p>\u30e2\u30c7\u30eb\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002\u6307\u6570\u95a2\u6570\u7684\u306b\u6e1b\u8870\u3059\u308b <a href=\"../optimizers/noam.html\">Noam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f7f\u7528\u3059\u308b\u5834\u5408\u306b\u5fc5\u8981\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set parameter groups for optimization. </p>\n": "<p>\u6700\u9069\u5316\u7528\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u6b21\u306b\u30de\u30b9\u30af\u3059\u308b\u3068\u3001\u30c8\u30fc\u30af\u30f3\u304c\u30de\u30b9\u30af\u3055\u308c\u3001\u5c06\u6765\u306e\u30c8\u30fc\u30af\u30f3\u304c\u898b\u3048\u306a\u304f\u306a\u308a\u307e\u3059</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u30de\u30b9\u30af\u306f\u6700\u521d\u306e\u547c\u3073\u51fa\u3057\u3067\u521d\u671f\u5316\u3055\u308c\u307e\u3059</p>\n",
|
||||
"<p>Total number of optimization steps for learning rate cosine decay </p>\n": "<p>\u5b66\u7fd2\u7387\u30b3\u30b5\u30a4\u30f3\u6e1b\u8870\u306e\u6700\u9069\u5316\u30b9\u30c6\u30c3\u30d7\u306e\u7dcf\u6570</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Transformer configurations </p>\n": "<p>\u5909\u5727\u5668\u69cb\u6210</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Use <a href=\"../optimizers/adam_warmup_cosine_decay.html\">cosine decay optimizer</a>. This is what GPT uses. </p>\n": "<p><a href=\"../optimizers/adam_warmup_cosine_decay.html\">\u30b3\u30b5\u30a4\u30f3\u6e1b\u8870\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f7f\u7528\u3057\u3066\u304f\u3060\u3055\u3044</a>\u3002\u3053\u308c\u304cGPT\u304c\u4f7f\u7528\u3059\u308b\u3082\u306e\u3067\u3059</p>\u3002\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<p>Weight decay </p>\n": "<p>\u4f53\u91cd\u6e1b\u5c11</p>\n",
|
||||
"<p>Weight decay is decoupled from gradients </p>\n": "<p>\u91cd\u91cf\u306e\u6e1b\u8870\u306f\u52fe\u914d\u304b\u3089\u5207\u308a\u96e2\u3055\u308c\u307e\u3059</p>\n",
|
||||
"<p>create the pytorch optimizer object </p>\n": "<p>pytorch \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u3092\u4f5c\u6210\u3059\u308b</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../models.html#Encoder\">\u5909\u5727\u5668\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span><a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u306f\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3059 (\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u4ed8\u304d)</a></li>\n</ul><li><span translate=no>_^_2_^_</span><a href=\"../models.html#Generator\">\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u6700\u5f8c\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u3067\u3059</a>\u3002</li>\n",
|
||||
"GPT": "GPT",
|
||||
"Implementation/tutorial of GPT model and training code.": "GPT\u30e2\u30c7\u30eb\u3068\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306e\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"<h1>GPT</h1>\n<p>This is a tutorial/implementation of <a href=\"https://openai.com/blog/better-language-models/\">OpenAI GPT architecture</a> in <a href=\"https://pytorch.org\">PyTorch</a>. We got a bunch of implementation details from <a href=\"https://github.com/karpathy/minGPT\">minGPT</a> by <a href=\"https://twitter.com/karpathy\">@karpathy</a>. This implementation also uses character tiny shakespeare dataset.</p>\n<p>GPT model is essentially a standard transformer with a few tweaks. GPT-2 and especially GPT-3 models are quite large and won't fit on a single GPU and will need model parallelism. This implementation doesn't even use data parallelism and is intended to be more of a tutorial.</p>\n<p>Main differences of this compared to a simple autoregressive transformer are the parameter initialization, weight decay, and learning rate schedule. For the transformer we reuse the <a href=\"../transformers/index.html\">existing labml/nn transformer implementation</a>.</p>\n<p>Here's a notebook for training a GPT model on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/gpt/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>GPT</h1>\n</a><p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u4e2d <a href=\"https://openai.com/blog/better-language-models/\">OpenAI GPT \u67b6\u6784\u7684\u6559\u7a0b/\u5b9e\u73b0\u3002<a href=\"https://twitter.com/karpathy\">@karpathy</a> \u4ece <a href=\"https://github.com/karpathy/minGPT\">MinGpt</a> \u90a3\u91cc\u5f97\u5230\u4e86\u5f88\u591a\u5b9e\u73b0\u7ec6\u8282\u3002\u6b64\u5b9e\u73b0\u8fd8\u4f7f\u7528\u89d2\u8272\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u3002</p>\n<p>GPT \u6a21\u578b\u672c\u8d28\u4e0a\u662f\u4e00\u4e2a\u6807\u51c6\u7684\u53d8\u538b\u5668\uff0c\u4f46\u6709\u4e00\u4e9b\u8c03\u6574\u3002GPT-2\uff0c\u5c24\u5176\u662f GPT-3 \u6a21\u578b\u975e\u5e38\u5927\uff0c\u4e0d\u9002\u5408\u5355\u4e2a GPU\uff0c\u9700\u8981\u6a21\u578b\u5e76\u884c\u5904\u7406\u3002\u6b64\u5b9e\u73b0\u751a\u81f3\u4e0d\u4f7f\u7528\u6570\u636e\u5e76\u884c\u6027\uff0c\u65e8\u5728\u66f4\u50cf\u662f\u4e00\u4e2a\u6559\u7a0b\u3002</p>\n\u4e0e@@ <p>\u7b80\u5355\u7684\u81ea\u56de\u5f52\u8f6c\u6362\u5668\u76f8\u6bd4\uff0c\u5176\u4e3b\u8981\u533a\u522b\u5728\u4e8e\u53c2\u6570\u521d\u59cb\u5316\u3001\u6743\u91cd\u8870\u51cf\u548c\u5b66\u4e60\u901f\u7387\u8c03\u5ea6\u3002\u5bf9\u4e8e\u53d8\u538b\u5668\uff0c\u6211\u4eec\u91cd\u7528\u4e86<a href=\"../transformers/index.html\">\u73b0\u6709\u7684 labml/nn \u53d8\u6362\u5668\u5b9e\u73b0</a>\u3002</p>\n<p>\u8fd9\u662f\u4e00\u672c\u7528\u4e8e\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3 GPT \u6a21\u578b\u7684\u7b14\u8bb0\u672c\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/gpt/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>GPT model</h2>\n<p>This consists of a token embedding layer, transformer encoder, and a final linear layer that gives token logits.</p>\n": "<h2>GPT \u578b\u53f7</h2>\n<p>\u8fd9\u5305\u62ec\u4ee4\u724c\u5d4c\u5165\u5c42\u3001\u53d8\u538b\u5668\u7f16\u7801\u5668\u548c\u7ed9\u51fa\u4ee4\u724c\u65e5\u5fd7\u7684\u6700\u7ec8\u7ebf\u6027\u5c42\u3002</p>\n",
|
||||
"<h3>Create custom optimizer with weight decay</h3>\n<p>This code is taken from <a href=\"https://github.com/karpathy/minGPT\">minGPT</a>. This applies weight decay only to weights of linear layers.</p>\n": "<h3>\u521b\u5efa\u5177\u6709\u6743\u91cd\u8870\u51cf\u7684\u81ea\u5b9a\u4e49\u4f18\u5316\u5668</h3>\n<p>\u6b64\u4ee3\u7801\u53d6\u81ea <a href=\"https://github.com/karpathy/minGPT\">MingPT</a>\u3002\u8fd9\u4ec5\u5c06\u6743\u91cd\u8870\u51cf\u5e94\u7528\u4e8e\u7ebf\u6027\u56fe\u5c42\u7684\u6743\u91cd\u3002</p>\n",
|
||||
"<h3>Initialize weights</h3>\n<p>Weights of linear layers and embedding layers are initialized to <span translate=no>_^_0_^_</span> instead of the default Xavier initialzation.</p>\n": "<h3>\u521d\u59cb\u5316\u6743\u91cd</h3>\n<p>\u7ebf\u6027\u5c42\u548c\u5d4c\u5165\u5c42\u7684\u6743\u91cd\u521d\u59cb\u5316\u4e3a\uff0c<span translate=no>_^_0_^_</span>\u800c\u4e0d\u662f\u9ed8\u8ba4\u7684 Xavier \u521d\u59cb\u5316\u3002</p>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create GPT model and initialize weights</p>\n": "<p>\u521b\u5efa GPT \u6a21\u578b\u5e76\u521d\u59cb\u5316\u6743\u91cd</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Apply custom weight initialization </p>\n": "<p>\u5e94\u7528\u81ea\u5b9a\u4e49\u6743\u91cd\u521d\u59cb\u5316</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Collect names of parameters to apply weight decay </p>\n": "<p>\u6536\u96c6\u53c2\u6570\u540d\u79f0\u4ee5\u5e94\u7528\u6743\u91cd\u8870\u51cf</p>\n",
|
||||
"<p>Create a <a href=\"../optimizers/configs.html#OptimizerConfigs\">configurable optimizer</a>, so that we can change these simply by passing a config dictionary. </p>\n": "<p>\u521b\u5efa\u4e00\u4e2a<a href=\"../optimizers/configs.html#OptimizerConfigs\">\u53ef\u914d\u7f6e\u7684\u4f18\u5316\u5668</a>\uff0c\u8fd9\u6837\u6211\u4eec\u5c31\u53ef\u4ee5\u901a\u8fc7\u4f20\u9012\u914d\u7f6e\u5b57\u5178\u6765\u66f4\u6539\u5b83\u4eec\u3002</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Create subsequent mask if mask is not initialized or if the size of the mask is different </p>\n": "<p>\u5982\u679c\u63a9\u7801\u672a\u521d\u59cb\u5316\u6216\u63a9\u7801\u5927\u5c0f\u4e0d\u540c\uff0c\u5219\u521b\u5efa\u540e\u7eed\u63a9\u7801</p>\n",
|
||||
"<p>Custom optimizer </p>\n": "<p>\u81ea\u5b9a\u4e49\u4f18\u5316\u5668</p>\n",
|
||||
"<p>GPT model </p>\n": "<p>GPT \u578b\u53f7</p>\n",
|
||||
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT \u4f7f\u7528 GELU \u6fc0\u6d3b\u8fdb\u884c\u4f4d\u7f6e\u660e\u667a\u524d\u9988</p>\n",
|
||||
"<p>GPT uses a maximum learning rate of <span translate=no>_^_0_^_</span>. </p>\n": "<p>GPT \u4f7f\u7528\u7684\u6700\u5927\u5b66\u4e60\u901f\u7387\u4e3a<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p>Get all the parameters </p>\n": "<p>\u83b7\u53d6\u6240\u6709\u53c2\u6570</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u83b7\u53d6\u65e5\u5fd7</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f7f\u7528\u4f4d\u7f6e\u7f16\u7801\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Initialize biases to <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5c06\u504f\u5dee\u521d\u59cb\u5316\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of tokens for wamup </p>\n": "<p>wamup \u7684\u4ee3\u5e01\u6570\u91cf</p>\n",
|
||||
"<p>Number of warmup optimization steps </p>\n": "<p>\u9884\u70ed\u4f18\u5316\u6b65\u9aa4\u6570</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Parameters that are not decayed </p>\n": "<p>\u672a\u8870\u51cf\u7684\u53c2\u6570</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c\uff08\u7b2c\u4e8c\u4e2a\u503c\u7528\u4e8e\u72b6\u6001\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u8bad\u7ec3\u5668\u4e5f\u4e0e RNN \u4e00\u8d77\u4f7f\u7528\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set default weight decay. This is not required since we set the weight decay in the parameter groups. </p>\n": "<p>\u8bbe\u7f6e\u9ed8\u8ba4\u6743\u91cd\u8870\u51cf\u3002\u8fd9\u4e0d\u662f\u5fc5\u9700\u7684\uff0c\u56e0\u4e3a\u6211\u4eec\u5728\u53c2\u6570\u7ec4\u4e2d\u8bbe\u7f6e\u4e86\u6743\u91cd\u8870\u51cf\u3002</p>\n",
|
||||
"<p>Set model embedding size, required if we use <a href=\"../optimizers/noam.html\">Noam optimizer</a> which has an exponential decay. </p>\n": "<p>\u8bbe\u7f6e\u6a21\u578b\u5d4c\u5165\u5927\u5c0f\uff0c\u5982\u679c\u6211\u4eec\u4f7f\u7528\u5177\u6709\u6307\u6570\u8870\u51cf\u7684 <a href=\"../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a>\uff0c\u5219\u9700\u8981\u8bbe\u7f6e\u6a21\u578b\u5d4c\u5165\u5927\u5c0f\u3002</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set parameter groups for optimization. </p>\n": "<p>\u8bbe\u7f6e\u53c2\u6570\u7ec4\u4ee5\u8fdb\u884c\u4f18\u5316\u3002</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Subsequent mask, will mask out tokens from seeing future tokens </p>\n": "<p>\u540e\u7eed\u7684\u63a9\u7801\uff0c\u5c06\u63a9\u76d6\u4ee4\u724c\u4ee5\u514d\u770b\u5230\u672a\u6765\u7684\u4ee3\u5e01</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>The mask will be initialized on the first call </p>\n": "<p>\u63a9\u7801\u5c06\u5728\u7b2c\u4e00\u6b21\u8c03\u7528\u65f6\u521d\u59cb\u5316</p>\n",
|
||||
"<p>Total number of optimization steps for learning rate cosine decay </p>\n": "<p>\u5b66\u4e60\u901f\u7387\u4f59\u5f26\u8870\u51cf\u7684\u4f18\u5316\u6b65\u9aa4\u603b\u6570</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Transformer configurations </p>\n": "<p>\u53d8\u538b\u5668\u914d\u7f6e</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
|
||||
"<p>Use <a href=\"../optimizers/adam_warmup_cosine_decay.html\">cosine decay optimizer</a>. This is what GPT uses. </p>\n": "<p>\u4f7f\u7528<a href=\"../optimizers/adam_warmup_cosine_decay.html\">\u4f59\u5f26\u8870\u51cf\u4f18\u5316\u5668</a>\u3002\u8fd9\u5c31\u662f GPT \u4f7f\u7528\u7684\u3002</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"<p>Weight decay </p>\n": "<p>\u4f53\u91cd\u8870\u51cf</p>\n",
|
||||
"<p>Weight decay is decoupled from gradients </p>\n": "<p>\u6743\u91cd\u8870\u51cf\u4e0e\u68af\u5ea6\u5206\u79bb</p>\n",
|
||||
"<p>create the pytorch optimizer object </p>\n": "<p>\u521b\u5efa pytorch \u4f18\u5316\u5668\u5bf9\u8c61</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668<a href=\"../models.html#Encoder\">\u7f16\u7801\u5668</a></li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c<a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u5d4c\u5165\u6a21\u5757\uff08\u5e26\u6709\u4f4d\u7f6e\u7f16\u7801\uff09</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u7ed9<a href=\"../models.html#Generator\">\u51fa logit \u7684\u6700\u540e\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42</a>\u3002</li></ul>\n",
|
||||
"GPT": "GPT",
|
||||
"Implementation/tutorial of GPT model and training code.": "GPT \u6a21\u578b\u548c\u8bad\u7ec3\u4ee3\u7801\u7684\u5b9e\u73b0/\u6559\u7a0b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"<h1>Hierarchical Transformers Are More Efficient Language Models</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2110.13711\">Hierarchical Transformers Are More Efficient Language Models</a>.</p>\n<p>This paper introduces a hierarchical transformer architecture to handle long sequences efficiently. The first half of the transformer layers down-sample tokens and the second half up-samples with direct skip connections between layers of the same resolution. This is a little similar to <a href=\"../../diffusion/ddpm/unet.html\">U-Net</a> for vision tasks.</p>\n<p>They try different up-sampling and down-sampling techniques and build a model with the best performing up and down-sampling techniques which they call the hourglass model.</p>\n<p>Here we have implemented the simplest up-sampling and down-sampling techniques for simplicity. We will consider adding more complex (and better performing) implementations later.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for the hourglass model.</p>\n": "<h1>\u968e\u5c64\u578b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u3088\u308a\u52b9\u7387\u7684\u306a\u8a00\u8a9e\u30e2\u30c7\u30eb</h1>\n<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/2110.13711\">\u968e\u5c64\u578b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u3088\u308a\u52b9\u7387\u7684\u306a\u8a00\u8a9e\u30e2\u30c7\u30eb</a>\u300d<a href=\"https://pytorch.org\">\u3068\u3044\u3046\u8ad6\u6587\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a>\u3002</p>\n<p>\u672c\u7a3f\u3067\u306f\u3001\u9577\u3044\u30b7\u30fc\u30b1\u30f3\u30b9\u3092\u52b9\u7387\u7684\u306b\u51e6\u7406\u3059\u308b\u305f\u3081\u306e\u968e\u5c64\u578b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30ec\u30a4\u30e4\u30fc\u306e\u524d\u534a\u306f\u30c8\u30fc\u30af\u30f3\u3092\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u3001\u5f8c\u534a\u306f\u540c\u3058\u89e3\u50cf\u5ea6\u306e\u30ec\u30a4\u30e4\u30fc\u9593\u3092\u76f4\u63a5\u30b9\u30ad\u30c3\u30d7\u63a5\u7d9a\u3057\u3066\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u307e\u3059\u3002<a href=\"../../diffusion/ddpm/unet.html\">\u3053\u308c\u306f\u30d3\u30b8\u30e7\u30f3\u30bf\u30b9\u30af\u7528\u306eU-Net\u306b\u5c11\u3057\u4f3c\u3066\u3044\u307e\u3059</a></p>\u3002\n<p>\u5f7c\u3089\u306f\u3055\u307e\u3056\u307e\u306a\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3068\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u624b\u6cd5\u3092\u8a66\u3057\u3001\u7802\u6642\u8a08\u30e2\u30c7\u30eb\u3068\u547c\u3070\u308c\u308b\u6700\u3082\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u306e\u9ad8\u3044\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u624b\u6cd5\u3068\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u624b\u6cd5\u3092\u4f7f\u7528\u3057\u3066\u30e2\u30c7\u30eb\u3092\u69cb\u7bc9\u3057\u307e\u3059\u3002</p>\n<p>\u3053\u3053\u3067\u306f\u3001\u308f\u304b\u308a\u3084\u3059\u304f\u3059\u308b\u305f\u3081\u306b\u3001\u6700\u3082\u5358\u7d14\u306a\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3068\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u624b\u6cd5\u3092\u5b9f\u88c5\u3057\u307e\u3057\u305f\u3002\u5f8c\u307b\u3069\u3001\u3088\u308a\u8907\u96d1\u306a (\u305d\u3057\u3066\u3088\u308a\u9ad8\u6027\u80fd\u306a) \u5b9f\u88c5\u3092\u8ffd\u52a0\u3059\u308b\u3053\u3068\u3092\u691c\u8a0e\u3057\u307e\u3059</p>\u3002\n<p><a href=\"experiment.html\">\u7802\u6642\u8a08\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059</a>\u3002</p>\n",
|
||||
"<h2>Hourglass model</h2>\n<p>This model recursively adds layers to the middle while shortening the sequence by down-sampling. The shortened sequence processed by another hourglass model is sandwiched between two normal transformer layers. (A transformer layer has a <a href=\"../mha.html\">self-attention layer</a> and a <a href=\"../feed_forward.html\">position-wise feed-forward layer</a>).</p>\n": "<h2>\u7802\u6642\u8a08\u30e2\u30c7\u30eb</h2>\n<p>\u3053\u306e\u30e2\u30c7\u30eb\u306f\u3001\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306b\u3088\u3063\u3066\u30b7\u30fc\u30b1\u30f3\u30b9\u3092\u77ed\u7e2e\u3057\u306a\u304c\u3089\u3001\u4e2d\u592e\u306b\u30ec\u30a4\u30e4\u30fc\u3092\u518d\u5e30\u7684\u306b\u8ffd\u52a0\u3057\u307e\u3059\u3002\u5225\u306e\u7802\u6642\u8a08\u30e2\u30c7\u30eb\u3067\u51e6\u7406\u3055\u308c\u305f\u77ed\u7e2e\u30b7\u30fc\u30b1\u30f3\u30b9\u306f\u30012\u3064\u306e\u901a\u5e38\u306e\u30c8\u30e9\u30f3\u30b9\u30ec\u30a4\u30e4\u30fc\u306e\u9593\u306b\u631f\u307e\u308c\u307e\u3059\u3002\uff08<a href=\"../mha.html\">\u30c8\u30e9\u30f3\u30b9\u5c64\u306b\u306f\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u5c64\u3068\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64\u304c\u3042\u308a\u307e\u3059</a><a href=\"../feed_forward.html\">\uff09</a></p>\u3002\n",
|
||||
"<h3>Average pool shortening</h3>\n<p>This down-samples by a given factor with average pooling</p>\n": "<h3>\u5e73\u5747\u7684\u306a\u30d7\u30fc\u30eb\u77ed\u7e2e</h3>\n<p>\u3053\u308c\u306f\u3001\u5e73\u5747\u30d7\u30fc\u30ea\u30f3\u30b0\u3092\u4f7f\u7528\u3057\u3066\u7279\u5b9a\u306e\u4fc2\u6570\u3067\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Generate auto-regressive mask</h3>\n": "<h3>\u81ea\u52d5\u56de\u5e30\u30de\u30b9\u30af\u3092\u751f\u6210</h3>\n",
|
||||
"<h3>Naive up-sampling</h3>\n<p>This up-samples by repeating</p>\n": "<h3>\u30ca\u30a4\u30fc\u30d6\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0</h3>\n<p>\u3053\u308c\u3092\u7e70\u308a\u8fd4\u3057\u3066\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u307e\u3059</p>\n",
|
||||
"<h3>Shift right operation</h3>\n<p>This shifts the sequence to the right by the given number of steps</p>\n": "<h3>\u53f3\u30b7\u30d5\u30c8\u64cd\u4f5c</h3>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001\u6307\u5b9a\u3057\u305f\u30b9\u30c6\u30c3\u30d7\u6570\u3060\u3051\u30b7\u30fc\u30b1\u30f3\u30b9\u304c\u53f3\u306b\u30b7\u30d5\u30c8\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h3>\ud83d\udea7 Attention based up-sampling</h3>\n<span translate=no>_^_0_^_</span><p>where <span translate=no>_^_1_^_</span></p>\n": "<h3>\ud83d\udea7 \u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d9\u30fc\u30b9\u306e\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0</h3>\n<span translate=no>_^_0_^_</span><p>\u3069\u3053 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<h3>\ud83d\udea7 Down-sampling with attention</h3>\n<span translate=no>_^_0_^_</span><p>where <span translate=no>_^_1_^_</span> is average pooling or linear pooling.</p>\n": "<h3>\ud83d\udea7 \u6ce8\u610f\u3057\u3066\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0</h3>\n<span translate=no>_^_0_^_</span><p>\u3053\u3053\u3067<span translate=no>_^_1_^_</span>\u3001\u306f\u5e73\u5747\u30d7\u30fc\u30ea\u30f3\u30b0\u304b\u7dda\u5f62\u30d7\u30fc\u30ea\u30f3\u30b0\u304b\u3067\u3059\u3002</p>\n",
|
||||
"<h3>\ud83d\udea7 Linear pooling for down-sampling</h3>\n<p>This concatenates the consecutive tokens embeddings that need to be merged and do a linear transformation to map it to the size of a single token embedding.</p>\n": "<h3>\ud83d\udea7 \u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u7528\u306e\u30ea\u30cb\u30a2\u30d7\u30fc\u30ea\u30f3\u30b0</h3>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001\u30de\u30fc\u30b8\u304c\u5fc5\u8981\u306a\u9023\u7d9a\u3057\u305f\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u304c\u9023\u7d50\u3055\u308c\u30011 \u3064\u306e\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u306e\u30b5\u30a4\u30ba\u306b\u5408\u308f\u305b\u3066\u7dda\u5f62\u5909\u63db\u304c\u884c\u308f\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<h3>\ud83d\udea7 Linear projection for up-sampling</h3>\n<p>Make a linear projection of dense token embeddings to a size of <span translate=no>_^_0_^_</span>.</p>\n": "<h3>\ud83d\udea7 \u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u7528\u306e\u30ea\u30cb\u30a2\u30d7\u30ed\u30b8\u30a7\u30af\u30b7\u30e7\u30f3</h3>\n<p>\u5bc6\u5ea6\u306e\u9ad8\u3044\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u3001\u306e\u30b5\u30a4\u30ba\u306b\u5408\u308f\u305b\u3066\u7dda\u5f62\u6295\u5f71\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><a href=\".. / feed_forward.html\">Position wise feed-forward layers</a> </p>\n": "<p><a href=\".. / feed_forward.html\">\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30ec\u30a4\u30e4\u30fc</a></p>\n",
|
||||
"<p><a href=\"../mha.html\">Multi-head attention layer</a> </p>\n": "<p><a href=\"../mha.html\">\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc</a></p>\n",
|
||||
"<p><a href=\"../utils.html\">Subsequent mask</a>, will mask out tokens from seeing future tokens </p>\n": "<p><a href=\"../utils.html\">\u6b21\u306b\u30de\u30b9\u30af\u3059\u308b\u3068</a>\u3001\u30c8\u30fc\u30af\u30f3\u304c\u30de\u30b9\u30af\u3055\u308c\u3001\u5c06\u6765\u306e\u30c8\u30fc\u30af\u30f3\u304c\u898b\u3048\u306a\u304f\u306a\u308a\u307e\u3059</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Auto-regressive mask </p>\n": "<p>\u81ea\u52d5\u56de\u5e30\u30de\u30b9\u30af</p>\n",
|
||||
"<p>Autoregressive mask </p>\n": "<p>\u81ea\u5df1\u56de\u5e30\u30de\u30b9\u30af</p>\n",
|
||||
"<p>Average pooling layer </p>\n": "<p>\u5e73\u5747\u30d7\u30fc\u30ea\u30f3\u30b0\u5c64</p>\n",
|
||||
"<p>Center transformer layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30bb\u30f3\u30bf\u30fc\u30c8\u30e9\u30f3\u30b9\u5c64 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Concatenate the zeros and truncate the right </p>\n": "<p>0 \u3092\u9023\u7d50\u3057\u3066\u53f3\u3092\u5207\u308a\u6368\u3066\u308b</p>\n",
|
||||
"<p>Create a mask if we haven't created or sizes have changed </p>\n": "<p>\u307e\u3060\u4f5c\u6210\u3057\u3066\u3044\u306a\u3044\u5834\u5408\u3084\u30b5\u30a4\u30ba\u304c\u5909\u66f4\u3055\u308c\u305f\u5834\u5408\u306f\u30de\u30b9\u30af\u3092\u4f5c\u6210\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Final transformer layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u7d42\u5909\u5727\u5668\u5c64 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>If the shift is <span translate=no>_^_0_^_</span> return the original </p>\n": "<p>\u30b7\u30d5\u30c8\u304c\u306e\u5834\u5408\u3001<span translate=no>_^_0_^_</span>\u5143\u306e\u72b6\u614b\u306b\u623b\u3059</p>\n",
|
||||
"<p>If there are no more shortening (middle of the hourglass) </p>\n": "<p>\u30b7\u30e7\u30fc\u30c8\u30cb\u30f3\u30b0\u304c\u306a\u304f\u306a\u3063\u305f\u3089 (\u7802\u6642\u8a08\u306e\u771f\u3093\u4e2d)</p>\n",
|
||||
"<p>If we are at the center of the hourglass, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7802\u6642\u8a08\u306e\u4e2d\u5fc3\u306b\u3044\u308b\u3068 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Initial transformer layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u521d\u671f\u5909\u5727\u5668\u5c64 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Insert another hourglass model recursively </p>\n": "<p>\u5225\u306e\u7802\u6642\u8a08\u30e2\u30c7\u30eb\u3092\u518d\u5e30\u7684\u306b\u633f\u5165</p>\n",
|
||||
"<p>Pooling layer accepts shape <span translate=no>_^_0_^_</span> so we permute axes. </p>\n": "<p><span translate=no>_^_0_^_</span>\u30d7\u30fc\u30ea\u30f3\u30b0\u5c64\u306f\u5f62\u72b6\u3092\u53d7\u3051\u5165\u308c\u308b\u306e\u3067\u3001\u8ef8\u3092\u4e26\u3079\u66ff\u3048\u307e\u3059\u3002</p>\n",
|
||||
"<p>Repeat across the sequence dimension </p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b9\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u5168\u4f53\u3067\u7e70\u308a\u8fd4\u3057\u307e\u3059</p>\n",
|
||||
"<p>Shifting and shortening <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b7\u30d5\u30c8\u3068\u30b7\u30e7\u30fc\u30c8\u30cb\u30f3\u30b0 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Shortening or the down-sampling layer. We use the simplest form - average pooling. The paper shows that attention based down sampling works best, which we haven't implemented yet. </p>\n": "<p>\u30b7\u30e7\u30fc\u30c8\u30cb\u30f3\u30b0\u307e\u305f\u306f\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u30ec\u30a4\u30e4\u30fc\u3002\u6700\u3082\u5358\u7d14\u306a\u5f62\u5f0f\u3001\u3064\u307e\u308a\u5e73\u5747\u30d7\u30fc\u30ea\u30f3\u30b0\u3092\u4f7f\u7528\u3057\u307e\u3059\u3002\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u6ce8\u610f\u306b\u57fa\u3065\u304f\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u304c\u6700\u3082\u52b9\u679c\u7684\u3067\u3042\u308b\u3053\u3068\u304c\u793a\u3055\u308c\u3066\u3044\u307e\u3059\u304c\u3001\u307e\u3060\u5b9f\u88c5\u3057\u3066\u3044\u307e\u305b\u3093\u3002</p>\n",
|
||||
"<p>The center layer is another transformer layer </p>\n": "<p>\u4e2d\u592e\u306e\u5c64\u306f\u5225\u306e\u5909\u5727\u5668\u5c64\u3067\u3059</p>\n",
|
||||
"<p>The final transformer layer after up-sampling </p>\n": "<p>\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u5f8c\u306e\u6700\u5f8c\u306e\u30c8\u30e9\u30f3\u30b9\u5c64</p>\n",
|
||||
"<p>The shortening factor <span translate=no>_^_0_^_</span> (or the down-sampling rate) </p>\n": "<p>\u77ed\u7e2e\u4fc2\u6570 <span translate=no>_^_0_^_</span> (\u307e\u305f\u306f\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u30ec\u30fc\u30c8)</p>\n",
|
||||
"<p>The transformer layer before down-sampling </p>\n": "<p>\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u524d\u306e\u30c8\u30e9\u30f3\u30b9\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Truncate the extra embeddings at the end </p>\n": "<p>\u6700\u5f8c\u306e\u4f59\u5206\u306a\u57cb\u3081\u8fbc\u307f\u306f\u5207\u308a\u6368\u3066\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Up-sample the shortened sequence and add a skip connection <span translate=no>_^_0_^_</span> </p>\n": "<p>\u77ed\u7e2e\u3055\u308c\u305f\u30b7\u30fc\u30b1\u30f3\u30b9\u3092\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u3001\u30b9\u30ad\u30c3\u30d7\u63a5\u7d9a\u3092\u8ffd\u52a0\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Up-sampling layer. We use naive up-sampling for simplicity and the paper shows attention based up sampling works better. </p>\n": "<p>\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u30ec\u30a4\u30e4\u30fc\u3002\u7c21\u7565\u5316\u306e\u305f\u3081\u306b\u30ca\u30a4\u30fc\u30d6\u306a\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3092\u4f7f\u7528\u3057\u3066\u304a\u308a\u3001\u8ad6\u6587\u3067\u306f\u6ce8\u610f\u306b\u57fa\u3065\u304f\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u65b9\u304c\u52b9\u679c\u7684\u3067\u3042\u308b\u3053\u3068\u304c\u793a\u3055\u308c\u3066\u3044\u307e\u3059</p>\u3002\n",
|
||||
"<p>We shift the tokens to the right by <span translate=no>_^_0_^_</span> steps to make sure information doesn't leak from the future tokens to past tokens as a result of down-sampling and up-sampling </p>\n": "<p>\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3068\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u7d50\u679c\u3001\u5c06\u6765\u306e\u30c8\u30fc\u30af\u30f3\u304b\u3089\u904e\u53bb\u306e\u30c8\u30fc\u30af\u30f3\u306b\u60c5\u5831\u304c\u6f0f\u308c\u306a\u3044\u3088\u3046\u306b\u3001<span translate=no>_^_0_^_</span>\u30c8\u30fc\u30af\u30f3\u3092\u6bb5\u968e\u7684\u306b\u53f3\u306b\u30b7\u30d5\u30c8\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Zeros to be appended to the left </p>\n": "<p>\u5de6\u306b\u30bc\u30ed\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>cannot be negative </p>\n": "<p>\u8ca0\u306e\u5024\u306b\u3059\u308b\u3053\u3068\u306f\u3067\u304d\u307e\u305b\u3093</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u5f62\u72b6\u306e\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u5f62\u304c\u5408\u3063\u3066\u3044\u308b <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of heads in <a href=\"../mha.html\">multi-head attention layers</a> </li>\n<li><span translate=no>_^_1_^_</span> is the size of the token embeddings </li>\n<li><span translate=no>_^_2_^_</span> is the dropout probability </li>\n<li><span translate=no>_^_3_^_</span> is the dimensionality of the hidden layer in <a href=\"../feed_forward.html\">position-wise feed-forward layers</a> </li>\n<li><span translate=no>_^_4_^_</span> is the list of shortening factors</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../mha.html\">\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30ec\u30a4\u30e4\u30fc\u5185\u306e\u30d8\u30c3\u30c9\u6570\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u306e\u30b5\u30a4\u30ba\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u8131\u843d\u78ba\u7387\u3067\u3059</li>\n<li><span translate=no>_^_3_^_</span><a href=\"../feed_forward.html\">\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64\u306b\u304a\u3051\u308b\u96a0\u308c\u5c64\u306e\u6b21\u5143\u3067\u3059</a></li>\n<li><span translate=no>_^_4_^_</span>\u77ed\u7e2e\u4fc2\u6570\u306e\u30ea\u30b9\u30c8\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of steps to shift by</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u30b7\u30d5\u30c8\u3059\u308b\u30b9\u30c6\u30c3\u30d7\u6570\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the shortening factor</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u77ed\u7e2e\u4fc2\u6570</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the tensor with embeddings before down-sampling </li>\n<li><span translate=no>_^_1_^_</span> is the tensor of higher density (to be up-sampled) representations</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30c0\u30a6\u30f3\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u524d\u306e\u57cb\u3081\u8fbc\u307f\u3092\u542b\u3080\u30c6\u30f3\u30bd\u30eb\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u3088\u308a\u9ad8\u3044\u5bc6\u5ea6\u306e (\u30a2\u30c3\u30d7\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u5bfe\u8c61\u306e) \u8868\u73fe\u306e\u30c6\u30f3\u30bd\u30eb\u3067\u3059</li></ul>\n",
|
||||
"Hierarchical Transformers Are More Efficient Language Models": "\u968e\u5c64\u578b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u3088\u308a\u52b9\u7387\u7684\u306a\u8a00\u8a9e\u30e2\u30c7\u30eb",
|
||||
"This is an annotated implementation/tutorial of hourglass model in PyTorch.": "\u3053\u308c\u306fPyTorch\u306e\u7802\u6642\u8a08\u30e2\u30c7\u30eb\u306e\u6ce8\u91c8\u4ed8\u304d\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"<h1>Hierarchical Transformers Are More Efficient Language Models</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2110.13711\">Hierarchical Transformers Are More Efficient Language Models</a>.</p>\n<p>This paper introduces a hierarchical transformer architecture to handle long sequences efficiently. The first half of the transformer layers down-sample tokens and the second half up-samples with direct skip connections between layers of the same resolution. This is a little similar to <a href=\"../../diffusion/ddpm/unet.html\">U-Net</a> for vision tasks.</p>\n<p>They try different up-sampling and down-sampling techniques and build a model with the best performing up and down-sampling techniques which they call the hourglass model.</p>\n<p>Here we have implemented the simplest up-sampling and down-sampling techniques for simplicity. We will consider adding more complex (and better performing) implementations later.</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for the hourglass model.</p>\n": "<h1>\u5206\u5c42\u8f6c\u6362\u5668\u662f\u66f4\u6709\u6548\u7684\u8bed\u8a00\u6a21\u578b</h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/2110.13711\">\u5206\u5c42\u8f6c\u6362\u5668\u662f\u66f4\u6709\u6548\u7684\u8bed\u8a00\u6a21\u578b\u300b\u7684</a> <a href=\"https://pytorch.org\">PyTorch</a> \u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u4ecb\u7ecd\u4e86\u4e00\u79cd\u5206\u5c42\u53d8\u538b\u5668\u67b6\u6784\uff0c\u53ef\u4ee5\u6709\u6548\u5730\u5904\u7406\u957f\u5e8f\u5217\u3002\u53d8\u538b\u5668\u5c42\u7684\u524d\u534a\u90e8\u5206\u5411\u4e0b\u91c7\u6837\u4ee4\u724c\uff0c\u540e\u534a\u90e8\u5206\u5728\u76f8\u540c\u5206\u8fa8\u7387\u7684\u5c42\u4e4b\u95f4\u4f7f\u7528\u76f4\u63a5\u8df3\u8fc7\u8fde\u63a5\u5411\u4e0a\u53d6\u6837\u3002\u8fd9\u4e0e\u7528\u4e8e\u89c6\u89c9\u4efb\u52a1\u7684 <a href=\"../../diffusion/ddpm/unet.html\">U-Net</a> \u6709\u70b9\u76f8\u4f3c\u3002</p>\n<p>\u4ed6\u4eec\u5c1d\u8bd5\u4e0d\u540c\u7684\u5411\u4e0a\u91c7\u6837\u548c\u5411\u4e0b\u91c7\u6837\u6280\u672f\uff0c\u5e76\u4f7f\u7528\u6027\u80fd\u6700\u4f73\u7684\u5411\u4e0a\u548c\u5411\u4e0b\u91c7\u6837\u6280\u672f\u6784\u5efa\u6a21\u578b\uff0c\u4ed6\u4eec\u79f0\u4e4b\u4e3a\u6c99\u6f0f\u6a21\u578b\u3002</p>\n\u4e3a\u7b80\u5355\u8d77@@ <p>\u89c1\uff0c\u6211\u4eec\u5728\u8fd9\u91cc\u5b9e\u73b0\u4e86\u6700\u7b80\u5355\u7684\u4e0a\u91c7\u6837\u548c\u5411\u4e0b\u91c7\u6837\u6280\u672f\u3002\u7a0d\u540e\u6211\u4eec\u4f1a\u8003\u8651\u6dfb\u52a0\u66f4\u590d\u6742\uff08\u6027\u80fd\u66f4\u597d\uff09\u7684\u5b9e\u73b0\u3002</p>\n<p>\u8fd9\u662f\u6c99\u6f0f\u6a21\u578b\u7684<a href=\"experiment.html\">\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
|
||||
"<h2>Hourglass model</h2>\n<p>This model recursively adds layers to the middle while shortening the sequence by down-sampling. The shortened sequence processed by another hourglass model is sandwiched between two normal transformer layers. (A transformer layer has a <a href=\"../mha.html\">self-attention layer</a> and a <a href=\"../feed_forward.html\">position-wise feed-forward layer</a>).</p>\n": "<h2>\u6c99\u6f0f\u578b\u53f7</h2>\n<p>\u8be5\u6a21\u578b\u9012\u5f52\u5730\u5728\u4e2d\u95f4\u6dfb\u52a0\u56fe\u5c42\uff0c\u540c\u65f6\u901a\u8fc7\u7f29\u51cf\u91c7\u6837\u6765\u7f29\u77ed\u5e8f\u5217\u3002\u7531\u53e6\u4e00\u4e2a\u6c99\u6f0f\u6a21\u578b\u5904\u7406\u7684\u7f29\u77ed\u5e8f\u5217\u5939\u5728\u4e24\u4e2a\u666e\u901a\u7684\u53d8\u538b\u5668\u5c42\u4e4b\u95f4\u3002\uff08\u53d8\u538b\u5668\u5c42\u5177\u6709<a href=\"../mha.html\">\u81ea\u6ce8\u610f\u529b\u5c42</a>\u548c<a href=\"../feed_forward.html\">\u4f4d\u7f6e\u524d\u9988\u5c42</a>\uff09\u3002</p>\n",
|
||||
"<h3>Average pool shortening</h3>\n<p>This down-samples by a given factor with average pooling</p>\n": "<h3>\u6c60\u5e73\u5747\u7f29\u77ed</h3>\n<p>\u8fd9\u4f1a\u6309\u7ed9\u5b9a\u56e0\u5b50\u5411\u4e0b\u91c7\u6837\uff0c\u5e76\u4f7f\u7528\u5e73\u5747\u6c47\u96c6</p>\n",
|
||||
"<h3>Generate auto-regressive mask</h3>\n": "<h3>\u751f\u6210\u81ea\u52a8\u56de\u5f52\u63a9\u7801</h3>\n",
|
||||
"<h3>Naive up-sampling</h3>\n<p>This up-samples by repeating</p>\n": "<h3>\u6734\u7d20\u7684\u5411\u4e0a\u91c7\u6837</h3>\n<p>\u8fd9\u901a\u8fc7\u91cd\u590d\u5411\u4e0a\u91c7\u6837</p>\n",
|
||||
"<h3>Shift right operation</h3>\n<p>This shifts the sequence to the right by the given number of steps</p>\n": "<h3>\u5411\u53f3\u79fb\u64cd\u4f5c</h3>\n<p>\u8fd9\u4f1a\u5c06\u5e8f\u5217\u5411\u53f3\u79fb\u52a8\u7ed9\u5b9a\u6b65\u6570</p>\n",
|
||||
"<h3>\ud83d\udea7 Attention based up-sampling</h3>\n<span translate=no>_^_0_^_</span><p>where <span translate=no>_^_1_^_</span></p>\n": "<h3>\ud83d\udea7 \u57fa\u4e8e\u6ce8\u610f\u529b\u7684\u5411\u4e0a\u91c7\u6837</h3>\n<span translate=no>_^_0_^_</span><p>\u5728\u54ea\u91cc<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<h3>\ud83d\udea7 Down-sampling with attention</h3>\n<span translate=no>_^_0_^_</span><p>where <span translate=no>_^_1_^_</span> is average pooling or linear pooling.</p>\n": "<h3>\ud83d\udea7 \u6ce8\u610f\u5411\u4e0b\u91c7\u6837</h3>\n<span translate=no>_^_0_^_</span><p>\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u5e73\u5747\u6c60\u5316\u6216\u7ebf\u6027\u6c60\u3002</p>\n",
|
||||
"<h3>\ud83d\udea7 Linear pooling for down-sampling</h3>\n<p>This concatenates the consecutive tokens embeddings that need to be merged and do a linear transformation to map it to the size of a single token embedding.</p>\n": "<h3>\ud83d\udea7 \u7528\u4e8e\u7f29\u51cf\u91c7\u6837\u7684\u7ebf\u6027\u6c60</h3>\n<p>\u8fd9\u5c06\u9700\u8981\u5408\u5e76\u7684\u8fde\u7eed\u4ee4\u724c\u5d4c\u5165\u8fde\u63a5\u8d77\u6765\uff0c\u5e76\u8fdb\u884c\u7ebf\u6027\u53d8\u6362\u4ee5\u5c06\u5176\u6620\u5c04\u5230\u5355\u4e2a\u4ee4\u724c\u5d4c\u5165\u7684\u5927\u5c0f\u3002</p>\n",
|
||||
"<h3>\ud83d\udea7 Linear projection for up-sampling</h3>\n<p>Make a linear projection of dense token embeddings to a size of <span translate=no>_^_0_^_</span>.</p>\n": "<h3>\ud83d\udea7 \u7528\u4e8e\u5411\u4e0a\u91c7\u6837\u7684\u7ebf\u6027\u6295\u5f71</h3>\n\u5c06@@ <p>\u5bc6\u96c6\u4ee4\u724c\u5d4c\u5165\u8fdb\u884c\u7ebf\u6027\u6295\u5f71\uff0c\u4f7f\u5176\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><a href=\".. / feed_forward.html\">Position wise feed-forward layers</a> </p>\n": "<p><a href=\".. / feed_forward.html\">\u5b9a\u4f4d\u660e\u667a\u7684\u524d\u9988\u5c42</a></p>\n",
|
||||
"<p><a href=\"../mha.html\">Multi-head attention layer</a> </p>\n": "<p><a href=\"../mha.html\">\u591a\u5934\u6ce8\u610f\u5c42</a></p>\n",
|
||||
"<p><a href=\"../utils.html\">Subsequent mask</a>, will mask out tokens from seeing future tokens </p>\n": "<p><a href=\"../utils.html\">\u540e\u7eed\u7684\u63a9\u7801</a>\uff0c\u5c06\u63a9\u76d6\u4ee4\u724c\u4ee5\u514d\u770b\u5230\u672a\u6765\u7684\u4ee3\u5e01</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Auto-regressive mask </p>\n": "<p>\u81ea\u52a8\u56de\u5f52\u63a9\u7801</p>\n",
|
||||
"<p>Autoregressive mask </p>\n": "<p>\u81ea\u56de\u5f52\u906e\u7f69</p>\n",
|
||||
"<p>Average pooling layer </p>\n": "<p>\u5e73\u5747\u6c60\u5c42</p>\n",
|
||||
"<p>Center transformer layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4e2d\u5fc3\u53d8\u538b\u5668\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Concatenate the zeros and truncate the right </p>\n": "<p>\u8fde\u63a5\u96f6\u5e76\u622a\u65ad\u53f3\u8fb9</p>\n",
|
||||
"<p>Create a mask if we haven't created or sizes have changed </p>\n": "<p>\u5982\u679c\u6211\u4eec\u5c1a\u672a\u521b\u5efa\u6216\u5927\u5c0f\u5df2\u66f4\u6539\uff0c\u8bf7\u521b\u5efa\u8499\u7248</p>\n",
|
||||
"<p>Final transformer layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u7ec8\u7684\u53d8\u538b\u5668\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>If the shift is <span translate=no>_^_0_^_</span> return the original </p>\n": "<p>\u5982\u679c\u79fb\u4f4d\u662f<span translate=no>_^_0_^_</span>\u8fd4\u56de\u539f\u6765\u7684</p>\n",
|
||||
"<p>If there are no more shortening (middle of the hourglass) </p>\n": "<p>\u5982\u679c\u6ca1\u6709\u66f4\u591a\u7684\u7f29\u77ed\uff08\u6c99\u6f0f\u7684\u4e2d\u95f4\uff09</p>\n",
|
||||
"<p>If we are at the center of the hourglass, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u6211\u4eec\u5728\u6c99\u6f0f\u7684\u4e2d\u5fc3<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Initial transformer layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u521d\u59cb\u53d8\u538b\u5668\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Insert another hourglass model recursively </p>\n": "<p>\u9012\u5f52\u5730\u63d2\u5165\u53e6\u4e00\u4e2a\u6c99\u6f0f\u6a21\u578b</p>\n",
|
||||
"<p>Pooling layer accepts shape <span translate=no>_^_0_^_</span> so we permute axes. </p>\n": "<p>\u6c60\u5316\u5c42\u63a5\u53d7\u5f62\u72b6<span translate=no>_^_0_^_</span>\uff0c\u6240\u4ee5\u6211\u4eec\u6392\u5217\u8f74\u3002</p>\n",
|
||||
"<p>Repeat across the sequence dimension </p>\n": "<p>\u5728\u5e8f\u5217\u7ef4\u5ea6\u4e0a\u91cd\u590d</p>\n",
|
||||
"<p>Shifting and shortening <span translate=no>_^_0_^_</span> </p>\n": "<p>\u79fb\u4f4d\u548c\u7f29\u77ed<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Shortening or the down-sampling layer. We use the simplest form - average pooling. The paper shows that attention based down sampling works best, which we haven't implemented yet. </p>\n": "<p>\u7f29\u77ed\u6216\u7f29\u51cf\u91c7\u6837\u5c42\u3002\u6211\u4eec\u4f7f\u7528\u6700\u7b80\u5355\u7684\u5f62\u5f0f\u2014\u2014\u5e73\u5747\u6c47\u96c6\u3002\u8be5\u8bba\u6587\u8868\u660e\uff0c\u57fa\u4e8e\u6ce8\u610f\u529b\u7684\u5411\u4e0b\u91c7\u6837\u6548\u679c\u6700\u597d\uff0c\u4f46\u6211\u4eec\u5c1a\u672a\u5b9e\u65bd\u3002</p>\n",
|
||||
"<p>The center layer is another transformer layer </p>\n": "<p>\u4e2d\u5fc3\u5c42\u662f\u53e6\u4e00\u4e2a\u53d8\u538b\u5668\u5c42</p>\n",
|
||||
"<p>The final transformer layer after up-sampling </p>\n": "<p>\u4e0a\u91c7\u6837\u540e\u7684\u6700\u7ec8\u53d8\u538b\u5668\u5c42</p>\n",
|
||||
"<p>The shortening factor <span translate=no>_^_0_^_</span> (or the down-sampling rate) </p>\n": "<p>\u7f29\u77ed\u7cfb\u6570<span translate=no>_^_0_^_</span>\uff08\u6216\u7f29\u51cf\u91c7\u6837\u7387\uff09</p>\n",
|
||||
"<p>The transformer layer before down-sampling </p>\n": "<p>\u4e0b\u91c7\u6837\u524d\u7684\u53d8\u538b\u5668\u5c42</p>\n",
|
||||
"<p>Truncate the extra embeddings at the end </p>\n": "<p>\u5728\u6700\u540e\u622a\u65ad\u591a\u4f59\u7684\u5d4c\u5165</p>\n",
|
||||
"<p>Up-sample the shortened sequence and add a skip connection <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5bf9\u7f29\u77ed\u7684\u5e8f\u5217\u8fdb\u884c\u5411\u4e0a\u91c7\u6837\u5e76\u6dfb\u52a0\u8df3\u8fc7\u8fde\u63a5<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Up-sampling layer. We use naive up-sampling for simplicity and the paper shows attention based up sampling works better. </p>\n": "<p>\u5411\u4e0a\u91c7\u6837\u56fe\u5c42\u3002\u4e3a\u4e86\u7b80\u5355\u8d77\u89c1\uff0c\u6211\u4eec\u4f7f\u7528\u5929\u771f\u7684\u5411\u4e0a\u91c7\u6837\uff0c\u672c\u6587\u663e\u793a\u57fa\u4e8e\u6ce8\u610f\u529b\u7684\u91c7\u6837\u6548\u679c\u66f4\u597d\u3002</p>\n",
|
||||
"<p>We shift the tokens to the right by <span translate=no>_^_0_^_</span> steps to make sure information doesn't leak from the future tokens to past tokens as a result of down-sampling and up-sampling </p>\n": "<p>\u6211\u4eec\u901a\u8fc7<span translate=no>_^_0_^_</span>\u6b65\u9aa4\u5c06\u4ee4\u724c\u5411\u53f3\u79fb\u52a8\uff0c\u4ee5\u786e\u4fdd\u4fe1\u606f\u4e0d\u4f1a\u56e0\u4e3a\u7f29\u51cf\u91c7\u6837\u548c\u4e0a\u91c7\u6837\u800c\u4ece\u672a\u6765\u7684\u4ee3\u5e01\u6cc4\u6f0f\u5230\u8fc7\u53bb\u7684\u4ee3\u5e01\u4e0a</p>\n",
|
||||
"<p>Zeros to be appended to the left </p>\n": "<p>\u8981\u8ffd\u52a0\u5230\u5de6\u8fb9\u7684\u96f6</p>\n",
|
||||
"<p>cannot be negative </p>\n": "<p>\u4e0d\u80fd\u4e3a\u8d1f\u6570</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u5f20\u91cf<span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u5f62\u72b6\u4e0d\u9519<span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of heads in <a href=\"../mha.html\">multi-head attention layers</a> </li>\n<li><span translate=no>_^_1_^_</span> is the size of the token embeddings </li>\n<li><span translate=no>_^_2_^_</span> is the dropout probability </li>\n<li><span translate=no>_^_3_^_</span> is the dimensionality of the hidden layer in <a href=\"../feed_forward.html\">position-wise feed-forward layers</a> </li>\n<li><span translate=no>_^_4_^_</span> is the list of shortening factors</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f<a href=\"../mha.html\">\u591a\u5934\u6ce8\u610f\u5c42\u4e2d\u7684\u5934\u90e8</a>\u6570\u91cf</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c\u5d4c\u5165\u7684\u5927\u5c0f</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u8f8d\u5b66\u6982\u7387</li>\n<li><span translate=no>_^_3_^_</span>\u662f<a href=\"../feed_forward.html\">\u4f4d\u7f6e\u524d\u9988\u5c42\u4e2d\u9690\u85cf\u5c42\u7684</a>\u7ef4\u5ea6</li>\n<li><span translate=no>_^_4_^_</span>\u662f\u7f29\u77ed\u56e0\u5b50\u6e05\u5355</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of steps to shift by</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8981\u79fb\u4f4d\u7684\u6b65\u6570</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the shortening factor</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u7f29\u77ed\u7cfb\u6570</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the tensor with embeddings before down-sampling </li>\n<li><span translate=no>_^_1_^_</span> is the tensor of higher density (to be up-sampled) representations</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5411\u4e0b\u91c7\u6837\u4e4b\u524d\u6709\u5d4c\u5165\u7684\u5f20\u91cf</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u8f83\u9ad8\u5bc6\u5ea6\uff08\u5f85\u5411\u4e0a\u91c7\u6837\uff09\u8868\u793a\u7684\u5f20\u91cf</li></ul>\n",
|
||||
"Hierarchical Transformers Are More Efficient Language Models": "\u5206\u5c42\u53d8\u6362\u5668\u662f\u66f4\u6709\u6548\u7684\u8bed\u8a00\u6a21\u578b",
|
||||
"This is an annotated implementation/tutorial of hourglass model in PyTorch.": "\u8fd9\u662f PyTorch \u4e2d\u6c99\u6f0f\u6a21\u578b\u7684\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0/\u6559\u7a0b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Hierarchical Transformers Are More Efficient Language Models</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">hourglass</a>.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h1><a href=\"index.html\">\u968e\u5c64\u578b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u3088\u308a\u52b9\u7387\u7684\u306a\u8a00\u8a9e\u30e2\u30c7\u30eb\u306e\u5b9f\u9a13</a></h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f\u3001\u7802\u6642\u8a08\u3092\u8a13\u7df4\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u306e PyTorch \u5b9f\u9a13\u3067\u3059\u3002</a></p>\n<p>\u3053\u308c\u306f\u3001<a href=\"../basic/autoregressive_experiment.html\">\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u5df1\u56de\u5e30NLP\u30bf\u30b9\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u69cb\u6210\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h2>Autoregressive language model</h2>\n": "<h2>\u81ea\u5df1\u56de\u5e30\u8a00\u8a9e\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u3001<a href=\"../basic/autoregressive_transformer.html\">\u5358\u7d14\u306a\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u81ea\u5df1\u56de\u5e30NLP\u30bf\u30b9\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3068\u69cb\u6210\u304b\u3089\u7d99\u627f\u3055\u308c\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create the model</p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u4f5c\u6210</p>\n",
|
||||
"<p><a href=\"../positional_encoding.html\">Fixed positional embeddings</a>.</p>\n<p>\ud83d\udcdd The <a href=\"https://github.com/google/trax/blob/master/trax/models/research/hourglass.py\">official paper implementation</a> use <a href=\"../xl/relative_mha.html\">relative attention</a> </p>\n": "<p><a href=\"../positional_encoding.html\">\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u3092\u4fee\u6b63\u3057\u307e\u3057\u305f</a>\u3002</p>\n<p>\ud83d\udcdd <a href=\"https://github.com/google/trax/blob/master/trax/models/research/hourglass.py\"><a href=\"../xl/relative_mha.html\">\u516c\u5f0f\u8ad6\u6587\u306e\u5b9f\u88c5\u306b\u306f\u6bd4\u8f03\u7684\u6ce8\u610f\u304c\u5fc5\u8981</a></a></p>\n",
|
||||
"<p><a href=\"index.html\">hourglass model</a> </p>\n": "<p><a href=\"index.html\">\u7802\u6642\u8a08\u30e2\u30c7\u30eb</a></p>\n",
|
||||
"<p>Add <a href=\"../positional_encoding.html\">positional embeddings</a> </p>\n": "<p><a href=\"../positional_encoding.html\">\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u3092\u8ffd\u52a0</a></p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create hourglass model </p>\n": "<p>\u7802\u6642\u8a08\u30e2\u30c7\u30eb\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create the auto-regressive wrapper </p>\n": "<p>\u81ea\u52d5\u56de\u5e30\u30e9\u30c3\u30d1\u30fc\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8131\u843d\u78ba\u7387</p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Final linear layer to predict the logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u4e88\u6e2c\u3059\u308b\u305f\u3081\u306e\u6700\u5f8c\u306e\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Get embeddings </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3092\u5165\u624b</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Hourglass </p>\n": "<p>\u7802\u6642\u8a08</p>\n",
|
||||
"<p>Model </p>\n": "<p>\u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u6570</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Return the logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u8fd4\u3059</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Shortening factors </p>\n": "<p>\u77ed\u7e2e\u8981\u56e0</p>\n",
|
||||
"<p>Size of feed-forward hidden layer </p>\n": "<p>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u96a0\u308c\u5c64\u306e\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>To normalize the final embeddings </p>\n": "<p>\u6700\u7d42\u7684\u306a\u57cb\u3081\u8fbc\u307f\u3092\u6b63\u898f\u5316\u3059\u308b\u306b\u306f</p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the tensor with token indexes of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u5f62\u72b6\u306e\u30c8\u30fc\u30af\u30f3\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u304c\u4ed8\u3044\u305f\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the vocabulary size </li>\n<li><span translate=no>_^_1_^_</span> is the size of the token embeddings </li>\n<li><span translate=no>_^_2_^_</span> is the dropout probability </li>\n<li><span translate=no>_^_3_^_</span> is the <a href=\"index.html\">hourglass model</a></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u306e\u30b5\u30a4\u30ba\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u8131\u843d\u78ba\u7387\u3067\u3059</li>\n<li><span translate=no>_^_3_^_</span><a href=\"index.html\">\u7802\u6642\u8a08\u30e2\u30c7\u30eb\u3067\u3059</a></li></ul>\n",
|
||||
"Hierarchical Transformers Are More Efficient Language Models Experiment": "\u968e\u5c64\u578b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306f\u3088\u308a\u52b9\u7387\u7684\u306a\u8a00\u8a9e\u30e2\u30c7\u30eb\u306e\u5b9f\u9a13",
|
||||
"This experiment trains a hourglass model on Tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067\u7802\u6642\u8a08\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Hierarchical Transformers Are More Efficient Language Models</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">hourglass</a>.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n<p><a href=\"https://app.labml.ai/run/855b82363e4911ec9ae4a5b9c69d5061\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">\u0db0\u0dd6\u0dbb\u0dcf\u0dc0\u0dbd\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc0\u0da9\u0dcf\u0dad\u0dca \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8 \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2</a> \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">\u0db4\u0dd0\u0dba \u0dc0\u0dd3\u0daf\u0dd4\u0dbb\u0dd4\u0dc0\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0dbb\u0dca\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p>\u0db8\u0dd9\u0dba\u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda <a href=\"../basic/autoregressive_experiment.html\">\u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba-\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0db8\u0dad \u0dba. </p>\n<p><a href=\"https://app.labml.ai/run/855b82363e4911ec9ae4a5b9c69d5061\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Autoregressive language model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba <a href=\"../basic/autoregressive_transformer.html\">\u0dc3\u0dbb\u0dbd \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba-\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d91\u0db1\u0dca\u0d91\u0dbd\u0dca\u0db4\u0dd3 \u0d9a\u0dbb\u0dca\u0dad\u0dc0\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4 \u0dc3\u0dc4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a>\u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0dda. </p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create the model</p>\n": "<p> \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p><a href=\"../positional_encoding.html\">Fixed positional embeddings</a>.</p>\n<p>\ud83d\udcdd The <a href=\"https://github.com/google/trax/blob/master/trax/models/research/hourglass.py\">official paper implementation</a> use <a href=\"../xl/relative_mha.html\">relative attention</a> </p>\n": "<p><a href=\"../positional_encoding.html\">\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca</a>. </p>\n<p>\ud83d\udcdd <a href=\"https://github.com/google/trax/blob/master/trax/models/research/hourglass.py\">\u0db1\u0dd2\u0dbd \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> <a href=\"../xl/relative_mha.html\">\u0dc3\u0dcf\u0db4\u0dda\u0d9a\u0dca\u0dc2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </p>\n",
|
||||
"<p><a href=\"index.html\">hourglass model</a> </p>\n": "<p><a href=\"index.html\">\u0db4\u0dd0\u0dba \u0dc0\u0dd3\u0daf\u0dd4\u0dbb\u0dd4 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</a> </p>\n",
|
||||
"<p>Add <a href=\"../positional_encoding.html\">positional embeddings</a> </p>\n": "<p><a href=\"../positional_encoding.html\">\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca</a> \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create hourglass model </p>\n": "<p>\u0db4\u0dd0\u0dba\u0dc0\u0dd3\u0daf\u0dd4\u0dbb\u0dd4 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create the auto-regressive wrapper </p>\n": "<p>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0daf\u0dc0\u0da7\u0db1\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u0d85\u0dad\u0dc4\u0dd0\u0dbb\u0daf\u0dd0\u0db8\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Final linear layer to predict the logits </p>\n": "<p>\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2 \u0db4\u0dc5 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Get embeddings </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Hourglass </p>\n": "<p>Hourglass </p>\n",
|
||||
"<p>Model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Return the logits </p>\n": "<p>\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca\u0d86\u0db4\u0dc3\u0dd4 \u0d91\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Shortening factors </p>\n": "<p>\u0d9a\u0dd9\u0da7\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0dcf\u0db0\u0d9a </p>\n",
|
||||
"<p>Size of feed-forward hidden layer </p>\n": "<p>\u0db4\u0ddd\u0dc2\u0d9a\u0d89\u0daf\u0dd2\u0dbb\u0dd2 \u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>To normalize the final embeddings </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca </p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the tensor with token indexes of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0daf\u0dbb\u0dca\u0dc1\u0d9a \u0dc3\u0dc4\u0dd2\u0dad \u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dba\u0dba\u0dd2 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the vocabulary size </li>\n<li><span translate=no>_^_1_^_</span> is the size of the token embeddings </li>\n<li><span translate=no>_^_2_^_</span> is the dropout probability </li>\n<li><span translate=no>_^_3_^_</span> is the <a href=\"index.html\">hourglass model</a></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0\u0dda \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dba\u0dd2 </li>\n<li><span translate=no>_^_1_^_</span> \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dba\u0dd2 </li>\n<li><span translate=no>_^_2_^_</span> \u0d85\u0dad\u0dc4\u0dd0\u0dbb \u0daf\u0dd0\u0db8\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </li>\n<li><span translate=no>_^_3_^_</span> <a href=\"index.html\">\u0db4\u0dd0\u0dba \u0d9c\u0dca\u0dbd\u0dcf\u0dc3\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dba\u0dd2</a></li></ul>\n",
|
||||
"Hierarchical Transformers Are More Efficient Language Models Experiment": "\u0db0\u0dd6\u0dbb\u0dcf\u0dc0\u0dbd\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc0\u0da9\u0dcf\u0dad\u0dca \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8 \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains a hourglass model on Tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba\u0dda \u0db4\u0dd0\u0dba \u0dc0\u0dd3\u0daf\u0dd4\u0dbb\u0dd4 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Hierarchical Transformers Are More Efficient Language Models</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">hourglass</a>.</p>\n<p>This is based on <a href=\"../basic/autoregressive_experiment.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h1><a href=\"index.html\">\u5206\u5c42\u8f6c\u6362\u5668\u662f\u66f4\u6709\u6548\u7684\u8bed\u8a00\u6a21\u578b</a>\u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3<a href=\"index.html\">\u6c99\u6f0f</a>\u3002</p>\n<p>\u8fd9\u662f\u57fa\u4e8e<a href=\"../basic/autoregressive_experiment.html\">\u7b80\u5355\u53d8\u6362\u5668\u81ea\u56de\u5f52 NLP \u4efb\u52a1\u7684\u8bad\u7ec3\u5faa\u73af\u548c\u914d\u7f6e</a>\u3002</p>\n",
|
||||
"<h2>Autoregressive language model</h2>\n": "<h2>\u81ea\u56de\u5f52\u8bed\u8a00\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../basic/autoregressive_transformer.html\">training loop and configurations for a simple transformer auto-regressive NLP task</a>.</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea<a href=\"../basic/autoregressive_transformer.html\">\u8bad\u7ec3\u5faa\u73af\u548c\u7b80\u5355\u53d8\u538b\u5668\u81ea\u56de\u5f52\u81ea\u7136\u8bed\u8a00\u5904\u7406\u4efb\u52a1\u7684\u914d\u7f6e</a>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create the model</p>\n": "<p>\u521b\u5efa\u6a21\u578b</p>\n",
|
||||
"<p><a href=\"../positional_encoding.html\">Fixed positional embeddings</a>.</p>\n<p>\ud83d\udcdd The <a href=\"https://github.com/google/trax/blob/master/trax/models/research/hourglass.py\">official paper implementation</a> use <a href=\"../xl/relative_mha.html\">relative attention</a> </p>\n": "<p><a href=\"../positional_encoding.html\">\u4fee\u590d\u4e86\u4f4d\u7f6e\u5d4c\u5165</a>\u3002</p>\n<p>\ud83d\udcdd <a href=\"https://github.com/google/trax/blob/master/trax/models/research/hourglass.py\">\u5b98\u65b9\u8bba\u6587\u5b9e\u65bd</a>\u4f7f\u7528<a href=\"../xl/relative_mha.html\">\u76f8\u5bf9\u5173\u6ce8</a></p>\n",
|
||||
"<p><a href=\"index.html\">hourglass model</a> </p>\n": "<p><a href=\"index.html\">\u6c99\u6f0f\u578b\u53f7</a></p>\n",
|
||||
"<p>Add <a href=\"../positional_encoding.html\">positional embeddings</a> </p>\n": "<p>\u6dfb\u52a0<a href=\"../positional_encoding.html\">\u4f4d\u7f6e\u5d4c\u5165</a></p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Create hourglass model </p>\n": "<p>\u521b\u5efa\u6c99\u6f0f\u6a21\u578b</p>\n",
|
||||
"<p>Create the auto-regressive wrapper </p>\n": "<p>\u521b\u5efa\u81ea\u52a8\u56de\u5f52\u5c01\u88c5</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8f8d\u5b66\u6982\u7387</p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Final linear layer to predict the logits </p>\n": "<p>\u9884\u6d4b\u5bf9\u6570\u7684\u6700\u7ec8\u7ebf\u6027\u5c42</p>\n",
|
||||
"<p>Get embeddings </p>\n": "<p>\u83b7\u53d6\u5d4c\u5165</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u83b7\u53d6\u65e5\u5fd7</p>\n",
|
||||
"<p>Hourglass </p>\n": "<p>\u6c99\u6f0f</p>\n",
|
||||
"<p>Model </p>\n": "<p>\u578b\u53f7</p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u6ce8\u610f\u5934\u6570\u91cf</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Return the logits </p>\n": "<p>\u8fd4\u56de\u65e5\u5fd7</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Shortening factors </p>\n": "<p>\u7f29\u77ed\u56e0\u7d20</p>\n",
|
||||
"<p>Size of feed-forward hidden layer </p>\n": "<p>\u524d\u9988\u9690\u85cf\u5c42\u7684\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>To normalize the final embeddings </p>\n": "<p>\u89c4\u8303\u5316\u6700\u7ec8\u5d4c\u5165</p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the tensor with token indexes of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u4ee4\u724c\u7d22\u5f15\u4e3a\u5f62\u72b6\u7684\u5f20\u91cf<span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the vocabulary size </li>\n<li><span translate=no>_^_1_^_</span> is the size of the token embeddings </li>\n<li><span translate=no>_^_2_^_</span> is the dropout probability </li>\n<li><span translate=no>_^_3_^_</span> is the <a href=\"index.html\">hourglass model</a></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u8bcd\u6c47\u91cf\u662f\u591a\u5c11</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c\u5d4c\u5165\u7684\u5927\u5c0f</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u8f8d\u5b66\u6982\u7387</li>\n<li><span translate=no>_^_3_^_</span>\u662f<a href=\"index.html\">\u6c99\u6f0f\u578b\u53f7\u5417</a></li></ul>\n",
|
||||
"Hierarchical Transformers Are More Efficient Language Models Experiment": "\u5206\u5c42\u53d8\u6362\u5668\u662f\u66f4\u6709\u6548\u7684\u8bed\u8a00\u6a21\u578b\u5b9e\u9a8c",
|
||||
"This experiment trains a hourglass model on Tiny Shakespeare dataset.": "\u8fd9\u4e2a\u5b9e\u9a8c\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u6c99\u6f0f\u6a21\u578b\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>k-Nearest Neighbor Language Models</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/1911.00172\">Generalization through Memorization: Nearest Neighbor Language Models</a>. It uses k-nearest neighbors to improve perplexity of autoregressive transformer models.</p>\n<p>An autoregressive language model estimates <span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is the token at step <span translate=no>_^_2_^_</span> and <span translate=no>_^_3_^_</span> is the context, <span translate=no>_^_4_^_</span>.</p>\n<p>This paper, improves <span translate=no>_^_5_^_</span> using a k-nearest neighbor search on key-value pairs <span translate=no>_^_6_^_</span>, with search key <span translate=no>_^_7_^_</span>. Here <span translate=no>_^_8_^_</span> is an embedding of the context <span translate=no>_^_9_^_</span>. The paper (and this implementation) uses the <strong>input to the feed-forward layer of the final layer of the transformer</strong> as <span translate=no>_^_10_^_</span>.</p>\n<p>We use <a href=\"https://github.com/facebookresearch/faiss\">FAISS</a> to index <span translate=no>_^_11_^_</span>.</p>\n<h3>Implementation</h3>\n<p>So to run <span translate=no>_^_12_^_</span>NN-LM we need to:</p>\n<ul><li><a href=\"train_model.html\">Train a transformer model</a> </li>\n<li><a href=\"build_index.html\">Build an index</a> of <span translate=no>_^_13_^_</span> </li>\n<li><a href=\"eval_knn.html\">Evaluate kNN-ML</a> using <span translate=no>_^_14_^_</span>NN seach on <span translate=no>_^_15_^_</span> with <span translate=no>_^_16_^_</span></li></ul>\n<p>This experiment uses a small dataset so that we can run this without using up a few hundred giga-bytes of disk space for the index.</p>\n<p>The official implementation of <span translate=no>_^_17_^_</span>NN-LM can be found <a href=\"https://github.com/urvashik/knnlm\">here</a>.</p>\n": "<h1>K \u8fd1\u90bb\u8bed\u8a00\u6a21\u578b</h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/1911.00172\">\u901a\u8fc7\u8bb0\u5fc6\u63a8\u5e7f\uff1a\u6700\u8fd1\u90bb\u8bed\u8a00\u6a21\u578b</a>\u300b\u7684 <a href=\"https://pytorch.org\">PyTorch</a> \u5b9e\u73b0\u3002\u5b83\u4f7f\u7528 k \u6700\u8fd1\u90bb\u6765\u6539\u5584\u81ea\u56de\u5f52\u53d8\u538b\u5668\u6a21\u578b\u7684\u56f0\u60d1\u5ea6\u3002</p>\n<p>\u81ea\u56de\u5f52\u8bed\u8a00\u6a21\u578b\u4f30\u8ba1<span translate=no>_^_0_^_</span>\uff0c\u6b65\u9aa4\u4e2d\u7684\u6807\u8bb0\u5728\u54ea\u91cc<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\uff0c<span translate=no>_^_3_^_</span>\u662f\u4e0a\u4e0b\u6587\uff0c<span translate=no>_^_4_^_</span>\u3002</p>\n<p>\u672c\u6587\u6539\u8fdb\u4e86<span translate=no>_^_5_^_</span>\u4f7f\u7528\u5e26\u641c\u7d22\u952e\u7684\u952e\u503c\u5bf9<span translate=no>_^_6_^_</span>\u4f7f\u7528 k \u6700\u8fd1\u90bb\u641c\u7d22\u7684\u529f\u80fd<span translate=no>_^_7_^_</span>\u3002<span translate=no>_^_8_^_</span>\u8fd9\u662f\u4e0a\u4e0b\u6587\u7684\u5d4c\u5165<span translate=no>_^_9_^_</span>\u3002\u672c\u6587\uff08\u4ee5\u53ca\u672c\u5b9e\u73b0\uff09\u4f7f\u7528<strong>\u53d8\u538b\u5668\u6700\u540e\u4e00\u5c42\u524d\u9988\u5c42\u7684\u8f93\u5165</strong>\u4f5c\u4e3a<span translate=no>_^_10_^_</span>\u3002</p>\n<p>\u6211\u4eec\u4f7f\u7528 <a href=\"https://github.com/facebookresearch/faiss\">FAISS</a> \u8fdb\u884c\u7d22\u5f15<span translate=no>_^_11_^_</span>\u3002</p>\n<h3>\u5b9e\u65bd</h3>\n<p>\u56e0\u6b64\uff0c\u8981\u8fd0\u884c<span translate=no>_^_12_^_</span> NN-LM\uff0c\u6211\u4eec\u9700\u8981\uff1a</p>\n<ul><li><a href=\"train_model.html\">\u8bad\u7ec3\u53d8\u538b\u5668\u6a21\u578b</a></li>\n<li><a href=\"build_index.html\">\u5efa\u7acb\u7d22\u5f15</a><span translate=no>_^_13_^_</span></li>\n<li>\u4f7f\u7528 <a href=\"eval_knn.html\">NN \u641c\u7d22\u6765\u8bc4\u4f30 k<span translate=no>_^_14_^_</span> nn-ML</a><span translate=no>_^_15_^_</span><span translate=no>_^_16_^_</span></li></ul>\n<p>\u8fd9\u4e2a\u5b9e\u9a8c\u4f7f\u7528\u4e86\u4e00\u4e2a\u5c0f\u6570\u636e\u96c6\uff0c\u8fd9\u6837\u6211\u4eec\u5c31\u53ef\u4ee5\u5728\u4e0d\u5360\u7528\u51e0\u767e\u5343\u5146\u5b57\u8282\u7684\u7d22\u5f15\u78c1\u76d8\u7a7a\u95f4\u7684\u60c5\u51b5\u4e0b\u8fd0\u884c\u5b83\u3002</p>\n<p><span translate=no>_^_17_^_</span>NN-LM \u7684\u5b98\u65b9\u5b9e\u73b0\u53ef\u4ee5<a href=\"https://github.com/urvashik/knnlm\">\u5728\u8fd9\u91cc</a>\u627e\u5230\u3002</p>\n",
|
||||
"This is a simple PyTorch implementation/tutorial of the paper Generalization through Memorization: Nearest Neighbor Language Models using FAISS. It runs a kNN model on the final transformer layer embeddings to improve the loss of transformer based language models. It's also great for domain adaptation without pre-training.": "\u8fd9\u662f\u8bba\u6587\u300a\u8bb0\u5fc6\u6cdb\u5316\uff1a\u4f7f\u7528FAISS\u7684\u6700\u8fd1\u90bb\u8bed\u8a00\u6a21\u578b\u300b\u7684\u7b80\u5355PyTorch\u5b9e\u73b0/\u6559\u7a0b\u3002\u5b83\u5728\u6700\u7ec8\u7684\u53d8\u538b\u5668\u5c42\u5d4c\u5165\u4e0a\u8fd0\u884ckNN\u6a21\u578b\uff0c\u4ee5\u6539\u5584\u57fa\u4e8e\u53d8\u538b\u5668\u7684\u8bed\u8a00\u6a21\u578b\u7684\u635f\u8017\u3002\u5b83\u4e5f\u975e\u5e38\u9002\u5408\u65e0\u9700\u9884\u5148\u8bad\u7ec3\u7684\u9886\u57df\u9002\u5e94\u3002",
|
||||
"k-Nearest Neighbor Language Models": "K \u8fd1\u90bb\u8bed\u8a00\u6a21\u578b"
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user