These are variants with gated hidden layers for the FFN as introduced in paper GLU Variants Improve Transformer. We have omitted the bias terms as specified in the paper.
\n": "\u3053\u308c\u3089\u306f\u3001\u7d19\u306eGLU\u30d0\u30ea\u30a2\u30f3\u30c8\u6539\u826f\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3067\u7d39\u4ecb\u3055\u308c\u3066\u3044\u308b\u3088\u3046\u306b\u3001FFN\u7528\u306e\u30b2\u30fc\u30c8\u96a0\u308c\u5c64\u3092\u5099\u3048\u305f\u30d0\u30ea\u30a2\u30f3\u30c8\u3067\u3059\u3002\u8ad6\u6587\u3067\u660e\u8a18\u3055\u308c\u3066\u3044\u308b\u30d0\u30a4\u30a2\u30b9\u7528\u8a9e\u306f\u7701\u7565\u3057\u3066\u3044\u307e\u3059
\u3002\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_ where _^_1_^_
\n": "_^_0_^_\u3069\u3053 _^_1_^_
\n", "Source embedding with fixed positional encodings
\n": "\u56fa\u5b9a\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30bd\u30fc\u30b9\u57cb\u3081\u8fbc\u307f
\n", "_^_0_^_ where _^_1_^_
\nIt was introduced in paper Gaussian Error Linear Units.
\n": "_^_0_^_\u3069\u3053 _^_1_^_
\n\n", "Source embedding with learned positional encodings
\n": "\u5b66\u7fd2\u3057\u305f\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30bd\u30fc\u30b9\u57cb\u3081\u8fbc\u307f
\n", "Source embedding without positional encodings
\n": "\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306a\u3057\u306e\u30bd\u30fc\u30b9\u57cb\u3081\u8fbc\u307f
\n", "_^_0_^_
\n": "_^_0_^_
\n", "Creates a Position-wise FeedForward Network defined in _^_0_^_.
\n": "\n\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u4f4d\u7f6e\u5358\u4f4d\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u4f5c\u6210\u3057\u307e\u3059\u3002_^_0_^_
\n", "\nThis defines configurations for a transformer. The configurations are calculate using option functions. These are lazy loaded and therefore only the necessary modules are calculated.
\n": "\n\u3053\u308c\u306f\u5909\u5727\u5668\u306e\u69cb\u6210\u3092\u5b9a\u7fa9\u3057\u307e\u3059\u3002\u69cb\u6210\u306f\u30aa\u30d7\u30b7\u30e7\u30f3\u95a2\u6570\u3092\u4f7f\u7528\u3057\u3066\u8a08\u7b97\u3055\u308c\u307e\u3059\u3002\u3053\u308c\u3089\u306f\u9045\u5ef6\u30ed\u30fc\u30c9\u3055\u308c\u308b\u305f\u3081\u3001\u5fc5\u8981\u306a\u30e2\u30b8\u30e5\u30fc\u30eb\u3060\u3051\u304c\u8a08\u7b97\u3055\u308c\u307e\u3059
\u3002\n", "Create feedforward layer configurations
\n": "\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64\u69cb\u6210\u306e\u4f5c\u6210
\n", "Decoder layer
\n": "\u30c7\u30b3\u30fc\u30c0\u30fc\u5c64
\n", "Decoder
\n": "\u30c7\u30b3\u30fc\u30c0\u30fc
\n", "Encoder layer
\n": "\u30a8\u30f3\u30b3\u30fc\u30c0\u5c64
\n", "Encoder
\n": "\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc
\n", "Initialize a feed forward network
\n": "\n", "Logit generator
\n": "\u30ed\u30b8\u30c3\u30c8\u30fb\u30b8\u30a7\u30cd\u30ec\u30fc\u30bf\u30fc
\n", "Target embedding with fixed positional encodings
\n": "\u56fa\u5b9a\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30bf\u30fc\u30b2\u30c3\u30c8\u57cb\u3081\u8fbc\u307f
\n", "Target embedding with learned positional encodings
\n": "\u5b66\u7fd2\u3057\u305f\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30bf\u30fc\u30b2\u30c3\u30c8\u57cb\u3081\u8fbc\u307f
\n", "Activation in position-wise feedforward layer
\n": "\u4f4d\u7f6e\u5358\u4f4d\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64\u3067\u306e\u6d3b\u6027\u5316
\n", "Configurable Feedforward Layer
\n": "\u8a2d\u5b9a\u53ef\u80fd\u306a\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64
\n", "Decoder layer
\n": "\u30c7\u30b3\u30fc\u30c0\u30fc\u5c64
\n", "Dropout probability
\n": "\u8131\u843d\u78ba\u7387
\n", "Embedding layer for source
\n": "\u30bd\u30fc\u30b9\u306e\u57cb\u3081\u8fbc\u307f\u30ec\u30a4\u30e4\u30fc
\n", "Embedding layer for target (for decoder)
\n": "\u30bf\u30fc\u30b2\u30c3\u30c8\u7528\u57cb\u3081\u8fbc\u307f\u30ec\u30a4\u30e4\u30fc (\u30c7\u30b3\u30fc\u30c0\u30fc\u7528)
\n", "Encoder consisting of multiple decoder layers
\n": "\u8907\u6570\u306e\u30c7\u30b3\u30fc\u30c0\u30fc\u5c64\u3067\u69cb\u6210\u3055\u308c\u308b\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc
\n", "Encoder consisting of multiple encoder layers
\n": "\u8907\u6570\u306e\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u5c64\u3067\u69cb\u6210\u3055\u308c\u308b\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc
\n", "Encoder layer
\n": "\u30a8\u30f3\u30b3\u30fc\u30c0\u5c64
\n", "Encoder-decoder
\n": "\u30a8\u30f3\u30b3\u30fc\u30c0/\u30c7\u30b3\u30fc\u30c0
\n", "Logit generator for prediction
\n": "\u4e88\u6e2c\u7528\u30ed\u30b8\u30c3\u30c8\u30fb\u30b8\u30a7\u30cd\u30ec\u30fc\u30bf\u30fc
\n", "Number of attention heads
\n": "\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u6570
\n", "Number of features in in the hidden layer
\n": "\u96a0\u308c\u30ec\u30a4\u30e4\u30fc\u306b\u542b\u307e\u308c\u308b\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u6570
\n", "Number of features in the embedding
\n": "\u57cb\u3081\u8fbc\u307f\u306b\u542b\u307e\u308c\u308b\u6a5f\u80fd\u306e\u6570
\n", "Number of layers
\n": "\u30ec\u30a4\u30e4\u30fc\u6570
\n", "Number of tokens in the source vocabulary (for token embeddings)
\n": "\u30bd\u30fc\u30b9\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u306e\u30c8\u30fc\u30af\u30f3\u6570 (\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u7528)
\n", "Number of tokens in the target vocabulary (to generate logits for prediction)
\n": "\u30bf\u30fc\u30b2\u30c3\u30c8\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u5185\u306e\u30c8\u30fc\u30af\u30f3\u306e\u6570 (\u4e88\u6e2c\u7528\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u305f\u3081)
\n", "Position-wise feedforward layer
\n": "\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64
\n", "Predefined GLU variants
\n": "\u5b9a\u7fa9\u6e08\u307f\u306e GLU \u30d0\u30ea\u30a2\u30f3\u30c8
\n", "The decoder memory attention
\n": "\u30c7\u30b3\u30fc\u30c0\u30e1\u30e2\u30ea\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3
\n", "The decoder self attention
\n": "\u30c7\u30b3\u30fc\u30c0\u30fc\u306e\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3
\n", "The encoder self attention
\n": "\u30a8\u30f3\u30b3\u30fc\u30c0\u306e\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3
\n", "Transformer embedding size
\n": "\u5909\u5727\u5668\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba
\n", "Whether the FFN layer should be gated
\n": "FFN \u30ec\u30a4\u30e4\u30fc\u3092\u30b2\u30fc\u30c8\u3059\u3079\u304d\u304b\u3069\u3046\u304b
\n", "Whether the first fully connected layer should have a learnable bias
\n": "\u6700\u521d\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u4ed8\u3051\u308b\u3079\u304d\u304b\u3069\u3046\u304b
\n", "Whether the fully connected layer for the gate should have a learnable bias
\n": "\u30b2\u30fc\u30c8\u306e\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u8a2d\u3051\u308b\u3079\u304d\u304b\u3069\u3046\u304b
\n", "Whether the second fully connected layer should have a learnable bias
\n": "2 \u756a\u76ee\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u306b\u5b66\u7fd2\u53ef\u80fd\u306a\u30d0\u30a4\u30a2\u30b9\u3092\u8a2d\u5b9a\u3059\u3079\u304d\u304b\u3069\u3046\u304b
\n", "Configurable Transformer Components": "\u8a2d\u5b9a\u53ef\u80fd\u306a\u5909\u5727\u5668\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8", "These are configurable components that can be re-used quite easily.": "\u3053\u308c\u3089\u306f\u8a2d\u5b9a\u53ef\u80fd\u306a\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8\u3067\u3001\u7c21\u5358\u306b\u518d\u5229\u7528\u3067\u304d\u307e\u3059\u3002" }