{ "

Noam Optimizer

\n

This is the PyTorch implementation of optimizer introduced in the paper Attention Is All You Need.

\n": "

Noam \u4f18\u5316\u5668

\n

\u8fd9\u662f\u300a\u6ce8\u610f\u5c31\u662f\u4f60\u6240\u9700\u8981\u7684\u300b\u4e00\u6587\u4e2d\u4ecb\u7ecd\u7684\u4f18\u5316\u5668\u7684 PyTorch \u5b9e\u73b0\u3002

\n", "

Noam Optimizer

\n

This class extends from Adam optimizer defined in _^_0_^_.

\n": "

Noam \u4f18\u5316\u5668

\n

\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 Adam \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684_^_0_^_\u3002

\n", "

Get learning-rate

\n

_^_0_^_ where _^_1_^_ is the number of warmup steps.

\n": "

\u83b7\u53d6\u5b66\u4e60\u7387

\n

_^_0_^_\u5176\u4e2d_^_1_^_\u662f\u9884\u70ed\u6b65\u9aa4\u7684\u6570\u91cf\u3002

\n", "

Initialize the optimizer

\n\n": "

\u521d\u59cb\u5316\u4f18\u5316\u5668

\n\n", "

Plot learning rate for different warmups and model sizes

\n

_^_0_^_

\n": "

\u7ed8\u5236\u4e0d\u540c\u9884\u70ed\u548c\u6a21\u578b\u5927\u5c0f\u7684\u5b66\u4e60\u901f\u7387

\n

_^_0_^_

\n", "

_^_0_^_

\n": "

_^_0_^_

\n", "Noam optimizer from Attention is All You Need paper": "\u300a\u6ce8\u610f\u5c31\u662f\u4f60\u6240\u9700\u8981\u7684 Noam Optimizer\u300b\u8bba\u6587", "This is a tutorial/implementation of Noam optimizer. Noam optimizer has a warm-up period and then an exponentially decaying learning rate.": "\u8fd9\u662f Noam \u4f18\u5316\u5668\u7684\u6559\u7a0b/\u5b9e\u73b0\u3002Noam \u4f18\u5316\u5668\u6709\u4e00\u4e2a\u9884\u70ed\u671f\uff0c\u7136\u540e\u5b66\u4e60\u7387\u5448\u6307\u6570\u7ea7\u8870\u51cf\u3002" }