This class extends from AMSAdam optimizer defined in _^_0_^_.
\n": "\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 AmsadAM \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684_^_0_^_\u3002
\n", "_^_3_^_
\n": "_^_3_^_
\n", "Scaled inverse chi-squared is the distribution of squared inverse of mean of _^_0_^_ normal distributions. _^_1_^_ where _^_2_^_.
\n": "\u7f29\u653e\u9006\u5361\u65b9\u662f_^_0_^_\u6b63\u6001\u5206\u5e03\u5747\u503c\u7684\u9006\u5e73\u65b9\u5206\u5e03\u3002_^_1_^_\u5728\u54ea\u91cc_^_2_^_\u3002
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_ is tractable when _^_1_^_. We are being a little more conservative since it's an approximated value
\n": "_^_0_^_\u4ec0\u4e48\u65f6\u5019\u662f\u53ef\u4ee5\u5904\u7406_^_1_^_\u7684\u3002\u6211\u4eec\u7a0d\u5fae\u4fdd\u5b88\u4e00\u70b9\uff0c\u56e0\u4e3a\u5b83\u662f\u8fd1\u4f3c\u503c
\n", "Adam optimizer sometimes converges to a bad local optima during the initial stages of the training; especially when training transformers. Researches use warmups to counter this; for the the initial training steps (warm-up stage) they use a low learning rate. This paper identifies the problem to be the high variance of adaptive learning rate during initial stages of training, and counters it using a new rectification term to reduce variance.
\n": "\u5728\u8bad\u7ec3\u7684\u521d\u59cb\u9636\u6bb5\uff0cAdam optimizer \u6709\u65f6\u4f1a\u6536\u655b\u5230\u7cdf\u7cd5\u7684\u5c40\u90e8\u6700\u4f73\u503c\uff1b\u5c24\u5176\u662f\u5728\u8bad\u7ec3\u53d8\u5f62\u91d1\u521a\u65f6\u3002\u7814\u7a76\u4f7f\u7528\u70ed\u8eab\u6765\u5e94\u5bf9\u8fd9\u79cd\u60c5\u51b5\uff1b\u5bf9\u4e8e\u6700\u521d\u7684\u8bad\u7ec3\u6b65\u9aa4\uff08\u70ed\u8eab\u9636\u6bb5\uff09\uff0c\u4ed6\u4eec\u4f7f\u7528\u8f83\u4f4e\u7684\u5b66\u4e60\u7387\u3002\u672c\u6587\u5c06\u95ee\u9898\u786e\u5b9a\u4e3a\u8bad\u7ec3\u521d\u59cb\u9636\u6bb5\u81ea\u9002\u5e94\u5b66\u4e60\u7387\u7684\u9ad8\u65b9\u5dee\uff0c\u5e76\u4f7f\u7528\u65b0\u7684\u6821\u6b63\u672f\u8bed\u6765\u51cf\u5c11\u65b9\u5dee\u3002
\n", "Bias correction term for _^_0_^_, _^_1_^_
\n": "\u504f\u5dee\u6821\u6b63\u672f\u8bed_^_0_^_\uff0c_^_1_^_
\n", "Calculate _^_0_^_ the number of optimizer steps
\n": "_^_0_^_\u8ba1\u7b97\u4f18\u5316\u5668\u6b65\u6570
\n", "Calculate weight decay
\n": "\u8ba1\u7b97\u4f53\u91cd\u8870\u51cf
\n", "Computation without optimization
\n": "\u65e0\u9700\u4f18\u5316\u7684\u8ba1\u7b97
\n", "Denominator _^_0_^_
\n": "\u5206\u6bcd_^_0_^_
\n", "From _^_0_^_ distribution we have,
\n": "\u4ece_^_0_^_\u5206\u53d1\u6765\u770b\uff0c
\n", "From above we have _^_0_^_ where _^_1_^_. Note that _^_2_^_ here is the standard deviation and different from _^_3_^_ for momentum.
\n": "\u4ece\u4e0a\u9762\u770b\uff0c\u6211\u4eec\u6709_^_0_^_\u54ea\u91cc_^_1_^_\u3002\u8bf7\u6ce8\u610f\uff0c_^_2_^_\u8fd9\u91cc\u662f\u6807\u51c6\u5dee\uff0c\u4e0e\u52a8_^_3_^_\u91cf\u4e0d\u540c\u3002
\n", "Get _^_0_^_ and _^_1_^_
\n": "\u83b7\u53d6_^_0_^_\u548c_^_1_^_
\n", "Get _^_0_^_ and _^_1_^_; i.e. _^_2_^_ and _^_3_^_ without bias correction
\n": "Get_^_0_^_ an_^_1_^_ d; \u5373_^_3_^_\u4e0d_^_2_^_\u8fdb\u884c\u504f\u5dee\u6821\u6b63
\n", "Get learning rate
\n": "\u83b7\u53d6\u5b66\u4e60\u7387
\n", "Here we are taking the simple moving average of the last _^_0_^_ gradients. _^_1_^_ satisfies the following,
\n": "\u8fd9\u91cc\u6211\u4eec\u53d6\u6700\u540e\u4e00\u4e2a_^_0_^_\u68af\u5ea6\u7684\u7b80\u5355\u79fb\u52a8\u5e73\u5747\u7ebf\u3002_^_1_^_\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff0c
\n", "If _^_0_^_ is intractable
\n": "\u5982\u679c_^_0_^_\u662f\u68d8\u624b\u7684
\n", "If _^_0_^_ is intractable do a SGD with momentum
\n": "\u5982\u679c_^_0_^_\u96be\u4ee5\u89e3\u51b3\uff0c\u90a3\u5c31\u7528\u52bf\u5934\u505a\u65b0\u52a0\u5761\u5143
\n", "In order to ensure that the adaptive learning rate _^_0_^_ has consistent variance, we rectify the variance with _^_1_^_
\n": "\u4e3a\u4e86\u786e\u4fdd\u81ea\u9002\u5e94\u5b66\u4e60\u7387_^_0_^_\u5177\u6709\u4e00\u81f4\u7684\u65b9\u5dee\uff0c\u6211\u4eec\u4f7f\u7528\u4ee5\u4e0b\u65b9\u6cd5\u6821\u6b63\u65b9\u5dee_^_1_^_
\n", "Let _^_0_^_ and _^_1_^_ be the functions to calculate momentum and adaptive learning rate. For Adam, they are
\n": "\u8ba9_^_0_^_\u548c_^_1_^_\u6210\u4e3a\u8ba1\u7b97\u52a8\u91cf\u548c\u81ea\u9002\u5e94\u5b66\u4e60\u901f\u7387\u7684\u51fd\u6570\u3002\u5bf9\u4e9a\u5f53\u6765\u8bf4\uff0c\u4ed6\u4eec\u662f
\n", "Perform RAdam update
\n": "\u6267\u884c raDAM \u66f4\u65b0
\n", "Step size _^_0_^_
\n": "\u6b65\u957f_^_0_^_
\n", "The distribution of exponential moving average can be approximated as a simple moving average.
\n": "\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u7684\u5206\u5e03\u53ef\u4ee5\u8fd1\u4f3c\u4e3a\u7b80\u5355\u79fb\u52a8\u5e73\u5747\u7ebf\u3002
\n", "The paper also evaluates two variance reduction mechanisms: Adam-2k: Only compute the adaptive learning rate (_^_0_^_ in Adam) during the first 2k steps, without changing parameters or calculating momentum (_^_1_^_). Adam-eps: Adam with large _^_2_^_.
\n": "\u672c\u6587\u8fd8\u8bc4\u4f30\u4e86\u4e24\u79cd\u65b9\u5dee\u7f29\u51cf\u673a\u5236\uff1aadam-2K\uff1a\u4ec5\u8ba1\u7b97\u524d 2k \u6b65\u957f\u7684\u81ea\u9002\u5e94\u5b66\u4e60\u7387\uff08_^_0_^_\u5728 Adam \u4e2d\uff09\uff0c\u800c\u4e0d\u66f4\u6539\u53c2\u6570\u6216\u8ba1\u7b97\u52a8\u91cf\uff08_^_1_^_)\u3002adam-eps\uff1aAdam \u5f88\u5927_^_2_^_\u3002
\n", "Therefore the variance is minimized at maximal _^_0_^_ which is _^_1_^_. Let the minimum variance be _^_2_^_
\n": "\u56e0\u6b64\uff0c\u65b9\u5dee\u6700\u5c0f\u5316_^_0_^_\u4e3a\u6700\u5927\u503c_^_1_^_\u3002\u8ba9\u6700\u5c0f\u65b9\u5dee\u4e3a_^_2_^_
\n", "They estimate _^_0_^_ based on first order expansion of _^_1_^_ \ud83e\udd2a I didn't get how it was derived.
\n": "\u4ed6\u4eec_^_0_^_\u6839\u636e\u4e00\u9636\u6269\u5f20\u4f30\u8ba1_^_1_^_ \ud83e\udd2a \u6211\u4e0d\u660e\u767d\u5b83\u662f\u5982\u4f55\u5f97\u51fa\u7684\u3002
\n", "They prove that variance of _^_0_^_ decreases with _^_1_^_ when _^_2_^_.
\n": "\u4ed6\u4eec\u8bc1\u660e\u4e86\u968f\u65f6\u95f4\u53d8\u5316\u7684\u53d8\u5316_^_0_^_\u800c_^_1_^_\u964d\u4f4e_^_2_^_\u3002
\n", "This gives,
\n": "\u8fd9\u7ed9\u4e86\uff0c
\n", "This implementation is based on the official implementation of the paper On the Variance of the Adaptive Learning Rate and Beyond.
\n": "\u8be5\u5b9e\u65bd\u57fa\u4e8e\u300a\u81ea\u9002\u5e94\u5b66\u4e60\u7387\u53ca\u4ee5\u540e\u7684\u5dee\u5f02\u300b\u4e00\u6587\u7684\u6b63\u5f0f\u5b9e\u65bd\u3002
\n", "Update parameters _^_0_^_
\n": "\u66f4\u65b0\u53c2\u6570_^_0_^_
\n", "We have implemented it in PyTorch as an extension to our AMSGrad implementation thus requiring only the modifications to be implemented.
\n": "\u6211\u4eec\u5df2\u7ecf\u5728 PyTorch \u4e2d\u5b9e\u73b0\u4e86\u5b83\uff0c\u4f5c\u4e3a\u6211\u4eec\u7684 AmsGrad \u5b9e\u73b0\u7684\u6269\u5c55\uff0c\u56e0\u6b64\u53ea\u9700\u8981\u5b9e\u65bd\u4fee\u6539\u5373\u53ef\u3002
\n", "We have
\n": "\u6211\u4eec\u6709
\n", "Whether to optimize the computation by combining scalar computations
\n": "\u662f\u5426\u901a\u8fc7\u7ec4\u5408\u6807\u91cf\u8ba1\u7b97\u6765\u4f18\u5316\u8ba1\u7b97
\n", "where _^_0_^_ is _^_1_^_ for _^_2_^_. Lt _^_3_^_ and step _^_4_^_ be _^_5_^_, and _^_6_^_ be the rectification term at step _^_7_^_.
\n": "\u5728_^_0_^_\u54ea_^_1_^_\u91cc_^_2_^_\u3002Lt_^_3_^_ and step_^_4_^_ be_^_5_^_\uff0c\u7136\u540e_^_6_^_\u6210\u4e3a step \u7684\u6574\u6539\u671f\u9650_^_7_^_\u3002
\n", "which gives, _^_0_^_
\n": "\u8fd9\u7ed9\u4e86\uff0c_^_0_^_
\n", "_^_0_^_": "_^_0_^_", "A simple PyTorch implementation/tutorial of RAdam optimizer.": "\u4e00\u4e2a\u7b80\u5355\u7684 PyTorch \u5b9e\u73b0/RadAM \u4f18\u5316\u5668\u6559\u7a0b\u3002", "Rectified Adam (RAdam) optimizer": "\u6821\u6b63\u4e9a\u5f53 (raDAM) \u4f18\u5316\u5668" }