This class extends from AMSAdam optimizer defined in _^_0_^_.
\n": "\u3053\u306e\u30af\u30e9\u30b9\u306f\u3001\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b AmSadam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u62e1\u5f35\u3057\u305f\u3082\u306e\u3067\u3059\u3002_^_0_^_
\n", "_^_3_^_
\n": "_^_3_^_
\n", "Scaled inverse chi-squared is the distribution of squared inverse of mean of _^_0_^_ normal distributions. _^_1_^_ where _^_2_^_.
\n": "\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3055\u308c\u305f\u9006\u30ab\u30a4\u4e8c\u4e57\u306f\u3001\u6b63\u898f\u5206\u5e03\u306e\u5e73\u5747\u306e\u4e8c\u4e57\u9006\u6570\u306e\u5206\u5e03\u3067\u3059\u3002_^_0_^__^_1_^_\u3069\u3053_^_2_^_\u3002
\n", "_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_ is tractable when _^_1_^_. We are being a little more conservative since it's an approximated value
\n": "_^_0_^__^_1_^_\u3069\u3093\u306a\u3068\u304d\u3067\u3082\u6271\u3044\u3084\u3059\u3044\u3002\u304a\u304a\u3088\u305d\u306e\u5024\u306a\u306e\u3067\u3001\u3082\u3046\u5c11\u3057\u4fdd\u5b88\u7684\u306b\u3057\u3066\u3044\u307e\u3059
\n", "Adam optimizer sometimes converges to a bad local optima during the initial stages of the training; especially when training transformers. Researches use warmups to counter this; for the the initial training steps (warm-up stage) they use a low learning rate. This paper identifies the problem to be the high variance of adaptive learning rate during initial stages of training, and counters it using a new rectification term to reduce variance.
\n": "\u30a2\u30c0\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306f\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u521d\u671f\u6bb5\u968e\u3001\u7279\u306b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u3066\u3044\u308b\u3068\u304d\u306b\u3001\u4e0d\u9069\u5207\u306a\u5c40\u6240\u6700\u9069\u5024\u306b\u53ce\u675f\u3059\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u3002\u7814\u7a76\u8005\u306f\u3053\u308c\u306b\u5bfe\u6297\u3059\u308b\u305f\u3081\u306b\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u3092\u4f7f\u3044\u307e\u3059\u3002\u6700\u521d\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b9\u30c6\u30c3\u30d7\uff08\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6bb5\u968e\uff09\u3067\u306f\u4f4e\u3044\u5b66\u7fd2\u7387\u3092\u4f7f\u3044\u307e\u3059\u3002\u672c\u7a3f\u3067\u306f\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u521d\u671f\u6bb5\u968e\u306b\u304a\u3051\u308b\u9069\u5fdc\u5b66\u7fd2\u7387\u306e\u3070\u3089\u3064\u304d\u304c\u5927\u304d\u3044\u3068\u3044\u3046\u554f\u984c\u3092\u7279\u5b9a\u3057\u3001\u5206\u6563\u3092\u6e1b\u3089\u3059\u305f\u3081\u306e\u65b0\u3057\u3044\u4fee\u6b63\u9805\u3092\u7528\u3044\u3066\u305d\u306e\u554f\u984c\u306b\u5bfe\u51e6\u3057\u3066\u3044\u307e\u3059
\u3002\n", "Bias correction term for _^_0_^_, _^_1_^_
\n": "_^_0_^_\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u7528\u8a9e _^_1_^_
\n", "Calculate _^_0_^_ the number of optimizer steps
\n": "\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u30b9\u30c6\u30c3\u30d7\u6570\u306e\u8a08\u7b97 _^_0_^_
\n", "Calculate weight decay
\n": "\u4f53\u91cd\u6e1b\u5c11\u306e\u8a08\u7b97
\n", "Computation without optimization
\n": "\u6700\u9069\u5316\u306a\u3057\u306e\u8a08\u7b97
\n", "Denominator _^_0_^_
\n": "\u5206\u6bcd _^_0_^_
\n", "From _^_0_^_ distribution we have,
\n": "_^_0_^_\u79c1\u305f\u3061\u304c\u6301\u3063\u3066\u3044\u308b\u30c7\u30a3\u30b9\u30c8\u30ea\u30d3\u30e5\u30fc\u30b7\u30e7\u30f3\u304b\u3089\u3001
\n", "From above we have _^_0_^_ where _^_1_^_. Note that _^_2_^_ here is the standard deviation and different from _^_3_^_ for momentum.
\n": "\u4e0a\u304b\u3089\u898b\u308b\u3068\u3001_^_0_^_\u5834\u6240\u304c\u308f\u304b\u308a\u307e\u3059_^_1_^_\u3002_^_2_^_\u3053\u308c\u306f\u6a19\u6e96\u504f\u5dee\u3067\u3042\u308a\u3001_^_3_^_\u904b\u52d5\u91cf\u3068\u306f\u7570\u306a\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044
\u3002\n", "Get _^_0_^_ and _^_1_^_
\n": "_^_0_^_\u53d6\u5f97\u3057\u3066 _^_1_^_
\n", "Get _^_0_^_ and _^_1_^_; i.e. _^_2_^_ and _^_3_^_ without bias correction
\n": "Get _^_0_^_ \u3068_^_1_^_; \u3064\u307e\u308a_^_2_^_\u3001_^_3_^_\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u306a\u3057
\n", "Get learning rate
\n": "\u5b66\u7fd2\u7387\u3092\u53d6\u5f97
\n", "Here we are taking the simple moving average of the last _^_0_^_ gradients. _^_1_^_ satisfies the following,
\n": "\u3053\u3053\u3067\u306f\u3001_^_0_^_\u6700\u5f8c\u306e\u52fe\u914d\u306e\u5358\u7d14\u79fb\u52d5\u5e73\u5747\u3092\u53d6\u3063\u3066\u3044\u307e\u3059\u3002_^_1_^_\u4ee5\u4e0b\u3092\u6e80\u305f\u3057\u3001
\n", "If _^_0_^_ is intractable
\n": "_^_0_^_\u6cbb\u308a\u306b\u304f\u3044\u5834\u5408
\n", "If _^_0_^_ is intractable do a SGD with momentum
\n": "_^_0_^_\u624b\u306b\u8ca0\u3048\u306a\u3044\u306a\u3089\u52e2\u3044\u3092\u3064\u3051\u3066SGD\u3092\u3084\u308a\u307e\u3057\u3087\u3046
\n", "In order to ensure that the adaptive learning rate _^_0_^_ has consistent variance, we rectify the variance with _^_1_^_
\n": "_^_0_^_\u9069\u5fdc\u578b\u5b66\u7fd2\u7387\u306e\u3070\u3089\u3064\u304d\u304c\u4e00\u8cab\u3057\u3066\u3044\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3059\u308b\u305f\u3081\u306b\u3001\u5dee\u7570\u3092\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u4fee\u6b63\u3057\u307e\u3059\u3002_^_1_^_
\n", "Let _^_0_^_ and _^_1_^_ be the functions to calculate momentum and adaptive learning rate. For Adam, they are
\n": "_^_1_^_\u904b\u52d5\u91cf\u3068\u9069\u5fdc\u5b66\u7fd2\u7387\u3092\u8a08\u7b97\u3059\u308b\u95a2\u6570\u3068\u3057\u307e\u3057\u3087\u3046_^_0_^_\u3002\u30a2\u30c0\u30e0\u306b\u3068\u3063\u3066\u3001\u5f7c\u3089\u306f
\n", "Perform RAdam update
\n": "RaDAM \u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u3092\u5b9f\u884c
\n", "Step size _^_0_^_
\n": "\u30b9\u30c6\u30c3\u30d7\u30b5\u30a4\u30ba _^_0_^_
\n", "The distribution of exponential moving average can be approximated as a simple moving average.
\n": "\u6307\u6570\u79fb\u52d5\u5e73\u5747\u306e\u5206\u5e03\u306f\u3001\u5358\u7d14\u306a\u79fb\u52d5\u5e73\u5747\u3068\u3057\u3066\u8fd1\u4f3c\u3067\u304d\u307e\u3059\u3002
\n", "The paper also evaluates two variance reduction mechanisms: Adam-2k: Only compute the adaptive learning rate (_^_0_^_ in Adam) during the first 2k steps, without changing parameters or calculating momentum (_^_1_^_). Adam-eps: Adam with large _^_2_^_.
\n": "\u3053\u306e\u8ad6\u6587\u3067\u306f\u30012\u3064\u306e\u5206\u6563\u524a\u6e1b\u30e1\u30ab\u30cb\u30ba\u30e0\u306b\u3064\u3044\u3066\u3082\u8a55\u4fa1\u3057\u3066\u3044\u307e\u3059\u3002Adam-2k\uff1a\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u5909\u66f4\u3057\u305f\u308a\u3001\u904b\u52d5\u91cf\u3092\u8a08\u7b97\u3057\u305f\u308a\u305b\u305a\u306b\u3001_^_0_^_\u6700\u521d\u306e2k\u30b9\u30c6\u30c3\u30d7\u3067\u306f\uff08Adam\u3067\uff09\u9069\u5fdc\u5b66\u7fd2\u7387\u306e\u307f\u3092\u8a08\u7b97\u3057\u307e\u3059\uff08\uff09\u3002_^_1_^_Adam-EPS: \u30a2\u30c0\u30e0\u30fb\u30a6\u30a3\u30ba\u30fb\u30e9\u30fc\u30b8\u30fb\u30a6\u30a3\u30ba\u30fb\u30e9\u30fc\u30b8
. _^_2_^_\n", "Therefore the variance is minimized at maximal _^_0_^_ which is _^_1_^_. Let the minimum variance be _^_2_^_
\n": "\u3057\u305f\u304c\u3063\u3066\u3001\u5206\u6563\u306f\u6700\u5927\u5024_^_0_^_\u3001\u3064\u307e\u308a\u3067\u6700\u5c0f\u5316\u3055\u308c\u307e\u3059\u3002_^_1_^_\u6700\u5c0f\u5206\u6563\u3092\u6b21\u306e\u5f0f\u306b\u3057\u307e\u3057\u3087\u3046 _^_2_^_
\n", "They estimate _^_0_^_ based on first order expansion of _^_1_^_ \ud83e\udd2a I didn't get how it was derived.
\n": "\u3069\u3046\u5c0e\u304d\u51fa\u3055\u308c\u305f\u306e\u304b\u308f\u304b\u3089\u306a\u304b\u3063\u305f _^_1_^_ \ud83e\udd2a _^_0_^_ \u306e\u4e00\u6b21\u5c55\u958b\u306b\u57fa\u3065\u3044\u3066\u898b\u7a4d\u3082\u3063\u3066\u3044\u307e\u3059\u3002
\n", "They prove that variance of _^_0_^_ decreases with _^_1_^_ when _^_2_^_.
\n": "_^_0_^_\u6642\u9593\u3068\u3068\u3082\u306b\u3070\u3089\u3064\u304d\u304c\u5c0f\u3055\u304f\u306a\u308b\u3053\u3068\u3092\u8a3c\u660e\u3057\u3066\u3044\u307e\u3059_^_1_^_\u3002_^_2_^_
\n", "This gives,
\n": "\u3053\u308c\u306b\u3088\u308a\u3001
\n", "This implementation is based on the official implementation of the paper On the Variance of the Adaptive Learning Rate and Beyond.
\n": "\u3053\u306e\u5b9f\u88c5\u306f\u3001\u300c\u9069\u5fdc\u5b66\u7fd2\u7387\u3068\u305d\u306e\u5f8c\u306e\u5dee\u7570\u306b\u95a2\u3059\u308b\u8ad6\u6587\u300d\u306e\u516c\u5f0f\u5b9f\u88c5\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059\u3002
\n", "Update parameters _^_0_^_
\n": "\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u66f4\u65b0 _^_0_^_
\n", "We have implemented it in PyTorch as an extension to our AMSGrad implementation thus requiring only the modifications to be implemented.
\n": "amsGrad\u5b9f\u88c5\u306e\u62e1\u5f35\u3068\u3057\u3066PyTorch\u306b\u5b9f\u88c5\u3057\u305f\u306e\u3067\u3001\u5b9f\u88c5\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b\u306e\u306f\u5909\u66f4\u3060\u3051\u3067\u3059\u3002
\n", "We have
\n": "\u79c1\u305f\u3061\u306f\u6301\u3063\u3066\u3044\u307e\u3059
\n", "Whether to optimize the computation by combining scalar computations
\n": "\u30b9\u30ab\u30e9\u30fc\u8a08\u7b97\u3092\u7d44\u307f\u5408\u308f\u305b\u3066\u8a08\u7b97\u3092\u6700\u9069\u5316\u3059\u308b\u304b\u3069\u3046\u304b
\n", "where _^_0_^_ is _^_1_^_ for _^_2_^_. Lt _^_3_^_ and step _^_4_^_ be _^_5_^_, and _^_6_^_ be the rectification term at step _^_7_^_.
\n": "_^_0_^__^_1_^_\u3069\u3053\u304c_^_2_^_._^_3_^__^_4_^_\u4e00\u6b69\u3092\u8e0f\u307f\u51fa\u3057\u3066_^_5_^_\u3001_^_6_^_\u6bb5\u968e\u7684\u306a\u4fee\u6b63\u9805\u306b\u306a\u308a\u306a\u3055\u3044
\u3002_^_7_^_\n", "which gives, _^_0_^_
\n": "\u3053\u308c\u306b\u3088\u308a\u3001_^_0_^_
\n", "_^_0_^_": "_^_0_^_", "A simple PyTorch implementation/tutorial of RAdam optimizer.": "RaDAM \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u7c21\u5358\u306a PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002", "Rectified Adam (RAdam) optimizer": "\u4fee\u6b63\u3055\u308c\u305f\u30a2\u30c0\u30e0 (RaDAM) \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc" }