These are variants with gated hidden layers for the FFN as introduced in paper GLU Variants Improve Transformer. We have omitted the bias terms as specified in the paper.
\n": "\u8fd9\u4e9b\u662f\u5728\u8bba\u6587 \u300a GLU Variants Improve Transformer \u300b\u4e2d\u5305\u542b\u7684\u5404\u79cd\u5e26\u95e8\u63a7\u9690\u85cf\u5c42\u7684 FFN \u53d8\u4f53\u3002\u6211\u4eec\u5df2\u6309\u7167\u8bba\u6587\u89c4\u5b9a\u7701\u7565\u4e86\u504f\u7f6e\u9879\u3002
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_ where _^_1_^_
\n": "_^_0_^_\u5176\u4e2d\uff0c_^_1_^_
\n", "Source embedding with fixed positional encodings
\n": "\u4f7f\u7528\u56fa\u5b9a\u4f4d\u7f6e\u7f16\u7801\u8fdb\u884c\u6e90\u5d4c\u5165
\n", "_^_0_^_ where _^_1_^_
\nIt was introduced in paper Gaussian Error Linear Units.
\n": "_^_0_^_\u5176\u4e2d\uff0c_^_1_^_
\n\u8fd9\u662f\u5728\u8bba\u6587\u300a Gaussian Error Linear Units \u300b\u4e2d\u4ecb\u7ecd\u7684\u3002
\n", "Source embedding with learned positional encodings
\n": "\u4f7f\u7528\u53ef\u5b66\u4e60\u7684\u4f4d\u7f6e\u7f16\u7801\u8fdb\u884c\u5d4c\u5165
\n", "Source embedding without positional encodings
\n": "\u6ca1\u6709\u4f4d\u7f6e\u7f16\u7801\u7684\u6e90\u5d4c\u5165
\n", "_^_0_^_
\n": "_^_0_^_
\n", "Creates a Position-wise FeedForward Network defined in _^_0_^_.
\n": "\n\u5728_^_0_^_\u4e2d\u5b9a\u4e49\u4e86\u4e00\u4e2a\u4f4d\u7f6e\u524d\u9988\u7f51\u7edc\u3002
\n", "\nThis defines configurations for a transformer. The configurations are calculate using option functions. These are lazy loaded and therefore only the necessary modules are calculated.
\n": "\n\u8fd9\u5b9a\u4e49\u4e86 Transformer \u7684\u914d\u7f6e\u3002\u8fd9\u4e9b\u914d\u7f6e\u662f\u901a\u8fc7\u53ef\u9009\u62e9\u7684\u51fd\u6570\u8fdb\u884c\u8ba1\u7b97\u7684\u3002\u5b83\u4eec\u662f\u60f0\u6027\u52a0\u8f7d\u7684\uff0c\u56e0\u6b64\u53ea\u6709\u5fc5\u8981\u7684\u6a21\u5757\u624d\u4f1a\u88ab\u8ba1\u7b97\u3002
\n", "Create feedforward layer configurations
\n": "\u521b\u5efa\u524d\u9988\u5c42\u914d\u7f6e
\n", "Decoder layer
\n": "\u89e3\u7801\u5668\u5c42
\n", "Decoder
\n": "\u89e3\u7801\u5668
\n", "Encoder layer
\n": "\u7f16\u7801\u5668\u5c42
\n", "Encoder
\n": "\u7f16\u7801\u5668
\n", "Initialize a feed forward network
\n": "\u521d\u59cb\u5316\u524d\u9988\u7f51\u7edc
\n", "Logit generator
\n": "Logit \u751f\u6210\u5668
\n", "Target embedding with fixed positional encodings
\n": "\u4f7f\u7528\u56fa\u5b9a\u4f4d\u7f6e\u7f16\u7801\u8fdb\u884c\u76ee\u6807\u5d4c\u5165
\n", "Target embedding with learned positional encodings
\n": "\u4f7f\u7528\u53ef\u5b66\u4e60\u7684\u4f4d\u7f6e\u7f16\u7801\u8fdb\u884c\u76ee\u6807\u5d4c\u5165
\n", "Activation in position-wise feedforward layer
\n": "\u4f4d\u7f6e\u524d\u9988\u5c42\u4e2d\u7684\u6fc0\u6d3b\u51fd\u6570
\n", "Configurable Feedforward Layer
\n": "\u53ef\u914d\u7f6e\u7684\u524d\u9988\u5c42
\n", "Decoder layer
\n": "\u89e3\u7801\u5668\u5c42
\n", "Dropout probability
\n": "Dropout \u7387
\n", "Embedding layer for source
\n": "\u6e90\u6570\u636e\u7684\u5d4c\u5165\u5c42
\n", "Embedding layer for target (for decoder)
\n": "\u76ee\u6807\u6570\u636e\u7684\u5d4c\u5165\u5c42\uff08\u7528\u4e8e\u89e3\u7801\u5668\uff09
\n", "Encoder consisting of multiple decoder layers
\n": "\u7531\u591a\u4e2a\u89e3\u7801\u5668\u5c42\u7ec4\u6210\u7684\u7f16\u7801\u5668
\n", "Encoder consisting of multiple encoder layers
\n": "\u7531\u591a\u4e2a\u7f16\u7801\u5668\u5c42\u7ec4\u6210\u7684\u7f16\u7801\u5668
\n", "Encoder layer
\n": "\u7f16\u7801\u5668\u5c42
\n", "Encoder-decoder
\n": "\u7f16\u7801\u5668-\u89e3\u7801\u5668
\n", "Logit generator for prediction
\n": "\u7528\u4e8e\u9884\u6d4b\u7684 Logit \u751f\u6210\u5668
\n", "Number of attention heads
\n": "\u6ce8\u610f\u529b\u5934\u6570\u91cf
\n", "Number of features in in the hidden layer
\n": "\u9690\u85cf\u5c42\u4e2d\u7684\u7279\u5f81\u6570\u91cf
\n", "Number of features in the embedding
\n": "\u5d4c\u5165\u7684\u7279\u5f81\u6570\u91cf
\n", "Number of layers
\n": "\u5c42\u6570
\n", "Number of tokens in the source vocabulary (for token embeddings)
\n": "\u6e90\u8bcd\u6c47\u8868\u4e2d\u7684 token \u6570\u91cf\uff08\u7528\u4e8e token \u5d4c\u5165\uff09
\n", "Number of tokens in the target vocabulary (to generate logits for prediction)
\n": "\u76ee\u6807\u8bcd\u6c47\u8868\u4e2d\u7684 token \u6570\u91cf\uff08\u7528\u4e8e\u751f\u6210\u9884\u6d4b\u7684 logits \uff09
\n", "Position-wise feedforward layer
\n": "\u4f4d\u7f6e\u524d\u9988\u5c42
\n", "Predefined GLU variants
\n": "\u9884\u5b9a\u4e49\u7684 GLU \u53d8\u4f53
\n", "The decoder memory attention
\n": "\u89e3\u7801\u5668\u8bb0\u5fc6\u4e0e\u6ce8\u610f\u529b
\n", "The decoder self attention
\n": "\u89e3\u7801\u5668\u81ea\u6ce8\u610f\u529b
\n", "The encoder self attention
\n": "\u7f16\u7801\u5668\u81ea\u6ce8\u610f\u529b
\n", "Transformer embedding size
\n": "Transformer \u5d4c\u5165\u5927\u5c0f
\n", "Whether the FFN layer should be gated
\n": "\u662f\u5426\u5e94\u5bf9 FFN \u5c42\u8fdb\u884c\u95e8\u63a7
\n", "Whether the first fully connected layer should have a learnable bias
\n": "\u7b2c\u4e00\u4e2a\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e
\n", "Whether the fully connected layer for the gate should have a learnable bias
\n": "\u95e8\u63a7\u7684\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e
\n", "Whether the second fully connected layer should have a learnable bias
\n": "\u7b2c\u4e8c\u4e2a\u5168\u8fde\u63a5\u5c42\u662f\u5426\u5177\u6709\u53ef\u5b66\u4e60\u7684\u504f\u7f6e
\n", "Configurable Transformer Components": "\u53ef\u914d\u7f6e Transformer \u7ec4\u4ef6", "These are configurable components that can be re-used quite easily.": "\u8fd9\u4e9b\u662f\u53ef\u914d\u7f6e\u7684\u7ec4\u4ef6\uff0c\u53ef\u4ee5\u5f88\u5bb9\u6613\u5730\u91cd\u590d\u4f7f\u7528\u3002" }