{
"<h1><a href=\"https://nn.labml.ai/transformers/xl/index.html\">Transformer XL</a></h1>\n<p>This is an implementation of <a href=\"https://arxiv.org/abs/1901.02860\">Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context</a> in <a href=\"https://pytorch.org\">PyTorch</a>.</p>\n<p>Transformer has a limited attention span, equal to the length of the sequence trained in parallel. All these positions have a fixed positional encoding. Transformer XL increases this attention span by letting each of the positions pay attention to precalculated past embeddings. For instance if the context length is <span translate=no>_^_0_^_</span>, it will keep the embeddings of all layers for previous batch of length <span translate=no>_^_1_^_</span> and feed them to current step. If we use fixed-positional encodings these pre-calculated embeddings will have the same positions as the current context. They introduce relative positional encoding, where the positional encodings are introduced at the attention calculation.</p>\n<p>Annotated implementation of relative multi-headed attention is in <a href=\"https://nn.labml.ai/transformers/xl/relative_mha.html\"><span translate=no>_^_2_^_</span></a>.</p>\n<p>Here's <a href=\"https://nn.labml.ai/transformers/xl/experiment.html\">the training code</a> and a notebook for training a transformer XL model on Tiny Shakespeare dataset.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/xl/experiment.ipynb\"><span translate=no>_^_3_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/xl/index.html\">\u53d8\u538b\u5668 XL</a></h1>\n<p>\u8fd9\u662f <a href=\"https://pytorch.org\">PyTorch \u4e2d Transfor</a> <a href=\"https://arxiv.org/abs/1901.02860\">mer-XL\uff1a\u8d85\u8d8a\u56fa\u5b9a\u957f\u5ea6\u4e0a\u4e0b\u6587\u7684\u4e13\u5fc3\u8bed\u8a00\u6a21\u578b</a>\u7684\u5b9e\u73b0\u3002</p>\n<p>Transformer \u7684\u6ce8\u610f\u529b\u8de8\u5ea6\u6709\u9650\uff0c\u7b49\u4e8e\u5e76\u884c\u8bad\u7ec3\u5e8f\u5217\u7684\u957f\u5ea6\u3002\u6240\u6709\u8fd9\u4e9b\u4f4d\u7f6e\u90fd\u6709\u56fa\u5b9a\u7684\u4f4d\u7f6e\u7f16\u7801\u3002Transformer XL \u901a\u8fc7\u8ba9\u6bcf\u4e2a\u4f4d\u7f6e\u5173\u6ce8\u8fc7\u53bb\u9884\u5148\u8ba1\u7b97\u7684\u5d4c\u5165\u6b21\u6570\uff0c\u4ece\u800c\u5ef6\u957f\u4e86\u8fd9\u79cd\u6ce8\u610f\u529b\u8de8\u5ea6\u3002\u4f8b\u5982\uff0c\u5982\u679c\u4e0a\u4e0b\u6587\u957f\u5ea6\u4e3a<span translate=no>_^_0_^_</span>\uff0c\u5b83\u5c06\u4fdd\u7559\u524d\u4e00\u6279\u957f\u5ea6\u7684\u6240\u6709\u5c42\u7684\u5d4c\u5165<span translate=no>_^_1_^_</span>\u5e76\u5c06\u5176\u9988\u9001\u5230\u5f53\u524d\u6b65\u9aa4\u3002\u5982\u679c\u6211\u4eec\u4f7f\u7528\u56fa\u5b9a\u4f4d\u7f6e\u7f16\u7801\uff0c\u8fd9\u4e9b\u9884\u5148\u8ba1\u7b97\u7684\u5d4c\u5165\u5c06\u4e0e\u5f53\u524d\u4e0a\u4e0b\u6587\u5177\u6709\u76f8\u540c\u7684\u4f4d\u7f6e\u3002\u5b83\u4eec\u5f15\u5165\u4e86\u76f8\u5bf9\u4f4d\u7f6e\u7f16\u7801\uff0c\u5176\u4e2d\u4f4d\u7f6e\u7f16\u7801\u662f\u5728\u6ce8\u610f\u529b\u8ba1\u7b97\u65f6\u5f15\u5165\u7684\u3002</p>\n<p>\u76f8\u5bf9\u591a\u5934\u6ce8\u610f\u529b\u7684\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0\u5df2\u7ecf\u5f00\u59cb<a href=\"https://nn.labml.ai/transformers/xl/relative_mha.html\"><span translate=no>_^_2_^_</span></a>\u4e86\u3002</p>\n<p>\u8fd9\u662f\u7528\u4e8e<a href=\"https://nn.labml.ai/transformers/xl/experiment.html\">\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3 transformer XL \u6a21\u578b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u548c\u7b14\u8bb0\u672c\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/xl/experiment.ipynb\"><span translate=no>_^_3_^_</span></a></p>\n",
"Transformer XL": "\u53d8\u538b\u5668 XL"
}
wehub-resource-sync