Here we train a transformer that uses Fuzzy Tiling Activation in the Feed-Forward Network. We use it for a language model and train it on Tiny Shakespeare dataset for demonstration.
\nHowever, this is probably not the ideal task for FTA, and we believe FTA is more suitable for modeling data with continuous variables.
\n": "\u5728\u8fd9\u91cc\uff0c\u6211\u4eec\u8bad\u7ec3\u4e00\u53f0\u5728\u524d\u9988\u7f51\u7edc\u4e2d\u4f7f\u7528\u6a21\u7cca\u5207\u7247\u6fc0\u6d3b\u7684\u53d8\u538b\u5668\u3002\u6211\u4eec\u5c06\u5176\u7528\u4f5c\u8bed\u8a00\u6a21\u578b\uff0c\u5e76\u5728\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u4e0a\u5bf9\u5176\u8fdb\u884c\u8bad\u7ec3\u4ee5\u8fdb\u884c\u6f14\u793a\u3002
\n\u4f46\u662f\uff0c\u5bf9\u4e8e FTA \u6765\u8bf4\uff0c\u8fd9\u53ef\u80fd\u4e0d\u662f\u7406\u60f3\u7684\u4efb\u52a1\uff0c\u6211\u4eec\u8ba4\u4e3a FTA \u66f4\u9002\u5408\u5bf9\u5177\u6709\u8fde\u7eed\u53d8\u91cf\u7684\u6570\u636e\u8fdb\u884c\u5efa\u6a21\u3002
\n", "This is an autoregressive transformer model that uses Feed-Forward Networks with (Fuzzy Tiling Activations)(index.html).
\n": "\u8fd9\u662f\u4e00\u4e2a\u81ea\u56de\u5f52\u53d8\u538b\u5668\u6a21\u578b\uff0c\u5b83\u4f7f\u7528\u524d\u9988\u7f51\u7edc\u548c\uff08\u6a21\u7cca\u5e73\u94fa\u6fc0\u6d3b\uff09\uff08index.html\uff09\u3002
\n", "This inherits from _^_0_^_
\n": "\u8fd9\u7ee7\u627f\u81ea _^_0_^_
\n", "\n": "\n", "
_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_ and _^_1_^_ for DeepNorm
\n": "_^_0_^_\u5bf9_^_1_^_\u4e8e DeepNorm
\n", "Activation function _^_0_^_
\n": "\u6fc0\u6d3b\u529f\u80fd_^_0_^_
\n", "Adam optimizer with no warmup
\n": "\u6ca1\u6709\u9884\u70ed\u7684 Adam \u4f18\u5316\u5668
\n", "Apply dropout
\n": "\u7533\u8bf7\u9000\u5b66
\n", "Batch size _^_0_^_
\n": "\u6279\u91cf\u5927\u5c0f_^_0_^_
\n", "Create FTA activation module
\n": "\u521b\u5efa FTA \u6fc0\u6d3b\u6a21\u5757
\n", "Create auto-regressive mask
\n": "\u521b\u5efa\u81ea\u52a8\u56de\u5f52\u906e\u7f69
\n", "Create configs
\n": "\u521b\u5efa\u914d\u7f6e
\n", "Create experiment
\n": "\u521b\u5efa\u5b9e\u9a8c
\n", "Create the transformer. We re-use _^_0_^_ and _^_1_^_ implementations.
\n": "\u521b\u5efa\u53d8\u538b\u5668\u3002\u6211\u4eec\u91cd\u590d\u4f7f\u7528_^_0_^_\u548c_^_1_^_\u5b9e\u73b0\u3002
\n", "Embedding size
\n": "\u5d4c\u5165\u5927\u5c0f
\n", "FTA
\n": "\u81ea\u8d38\u533a
\n", "Feed forward layer size
\n": "\u524d\u9988\u56fe\u5c42\u5927\u5c0f
\n", "Get logits
\n": "\u83b7\u53d6\u65e5\u5fd7
\n", "Get the token embeddings
\n": "\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165
\n", "Hidden layer dropout
\n": "\u9690\u85cf\u56fe\u5c42\u4e22\u5931
\n", "Layer one parameterized by weight _^_0_^_ and bias _^_1_^_
\n": "\u7b2c\u4e00\u5c42\u6309\u6743\u91cd_^_0_^_\u548c\u504f\u5dee\u8fdb\u884c\u53c2\u6570\u5316_^_1_^_
\n", "Layer two parameterized by weight _^_0_^_ and bias _^_1_^_
\n": "\u7b2c\u4e8c\u5c42\u6309\u6743\u91cd_^_0_^_\u548c\u504f\u5dee\u8fdb\u884c\u53c2\u6570\u5316_^_1_^_
\n", "Model
\n": "\u578b\u53f7
\n", "Move to the device
\n": "\u79fb\u5230\u8bbe\u5907
\n", "Number of heads in the attention
\n": "\u5173\u6ce8\u7684\u5934\u90e8\u6570\u91cf
\n", "Number of layers
\n": "\u5c42\u6570
\n", "Override configurations
\n": "\u8986\u76d6\u914d\u7f6e
\n", "Prompt separator is blank
\n": "\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a
\n", "Readout layer
\n": "\u8bfb\u51fa\u5c42
\n", "Return results
\n": "\u8fd4\u56de\u7ed3\u679c
\n", "Run training
\n": "\u8dd1\u6b65\u8bad\u7ec3
\n", "Set model(s) for saving and loading
\n": "\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b
\n", "Size of each attention head
\n": "\u6bcf\u4e2a\u6ce8\u610f\u5934\u7684\u5927\u5c0f
\n", "Start the experiment
\n": "\u5f00\u59cb\u5b9e\u9a8c
\n", "Starting prompt for sampling
\n": "\u5f00\u59cb\u91c7\u6837\u63d0\u793a
\n", "Subsequent mask, will mask out tokens from seeing future tokens
\n": "\u540e\u7eed\u7684\u63a9\u7801\uff0c\u5c06\u63a9\u76d6\u4ee4\u724c\u4ee5\u514d\u770b\u5230\u672a\u6765\u7684\u4ee3\u5e01
\n", "Switch between training and validation for _^_0_^_ times per epoch
\n": "\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684_^_0_^_\u6b21\u6570
\n", "The mask will be initialized on the first call
\n": "\u63a9\u7801\u5c06\u5728\u7b2c\u4e00\u6b21\u8c03\u7528\u65f6\u521d\u59cb\u5316
\n", "Token embedding layer
\n": "\u4ee4\u724c\u5d4c\u5165\u5c42
\n", "Train for 32 epochs
\n": "\u8bad\u7ec3 32 \u4e2a\u65f6\u4ee3
\n", "Transformer encoder
\n": "\u53d8\u538b\u5668\u7f16\u7801
\n", "Transformer with _^_0_^_ layers
\n": "\u5e26_^_0_^_\u5c42\u7684\u53d8\u538b\u5668
\n", "Use Tiny Shakespeare dataset
\n": "\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6
\n", "Use a context size of _^_0_^_
\n": "\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a_^_0_^_
\n", "Use character level tokenizer
\n": "\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668
\n", "