This is a PyTorch implementation of the paper On the Convergence of Adam and Beyond.
\nWe implement this as an extension to our Adam optimizer implementation. The implementation it self is really small since it's very similar to Adam.
\nWe also have an implementation of the synthetic example described in the paper where Adam fails to converge.
\n": "\u8fd9\u662f PyTorch \u5bf9\u300a\u4e9a\u5f53\u4e0e\u8d85\u8d8a\u7684\u878d\u5408\u300b\u4e00\u6587\u7684\u5b9e\u73b0\u3002
\n\u6211\u4eec\u5c06\u5176\u4f5c\u4e3a\u6211\u4eec\u7684 Adam \u4f18\u5316\u5668\u5b9e\u73b0\u7684\u6269\u5c55\u3002\u5b83\u81ea\u8eab\u7684\u5b9e\u73b0\u975e\u5e38\u5c0f\uff0c\u56e0\u4e3a\u5b83\u4e0e\u4e9a\u5f53\u975e\u5e38\u76f8\u4f3c\u3002
\n\u6211\u4eec\u8fd8\u5b9e\u73b0\u4e86\u672c\u6587\u4e2d\u63cf\u8ff0\u7684\u5408\u6210\u793a\u4f8b\uff0c\u5176\u4e2d\u4e9a\u5f53\u672a\u80fd\u6536\u655b\u3002
\n", "This class extends from Adam optimizer defined in _^_0_^_. Adam optimizer is extending the class _^_1_^_ defined in _^_2_^_.
\n": "\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 Adam \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684_^_0_^_\u3002Adam \u4f18\u5316\u5668\u6b63\u5728\u6269\u5c55\u4e2d_^_1_^_\u5b9a\u4e49\u7684\u7c7b_^_2_^_\u3002
\n", "This is the synthetic experiment described in the paper, that shows a scenario where Adam fails.
\nThe paper (and Adam) formulates the problem of optimizing as minimizing the expected value of a function, _^_0_^_ with respect to the parameters _^_1_^_. In the stochastic training setting we do not get hold of the function _^_2_^_ it self; that is, when you are optimizing a NN _^_3_^_ would be the function on entire batch of data. What we actually evaluate is a mini-batch so the actual function is realization of the stochastic _^_4_^_. This is why we are talking about an expected value. So let the function realizations be _^_5_^_ for each time step of training.
\nWe measure the performance of the optimizer as the regret, _^_6_^_ where _^_7_^_ is the parameters at time step _^_8_^_, and _^_9_^_ is the optimal parameters that minimize _^_10_^_.
\nNow lets define the synthetic problem,
\n_^_11_^_where _^_12_^_. The optimal solution is _^_13_^_.
\nThis code will try running Adam and AMSGrad on this problem.
\n": "\u8fd9\u662f\u8bba\u6587\u4e2d\u63cf\u8ff0\u7684\u5408\u6210\u5b9e\u9a8c\uff0c\u5b83\u663e\u793a\u4e86\u4e9a\u5f53\u5931\u8d25\u7684\u60c5\u666f\u3002
\n\u672c\u6587\uff08\u548c\u4e9a\u5f53\uff09\u5c06\u4f18\u5316\u95ee\u9898\u63cf\u8ff0\u4e3a\u6700\u5c0f\u5316\u51fd\u6570_^_0_^_\u76f8\u5bf9\u4e8e\u53c2\u6570\u7684\u9884\u671f\u503c_^_1_^_\u3002\u5728\u968f\u673a\u8bad\u7ec3\u8bbe\u7f6e\u4e2d\uff0c\u6211\u4eec\u65e0\u6cd5\u638c\u63e1_^_2_^_\u5b83\u81ea\u8eab\u7684\u51fd\u6570\uff1b\u4e5f\u5c31\u662f\u8bf4\uff0c\u5f53\u4f60\u4f18\u5316\u65f6\uff0cNN_^_3_^_ \u5c06\u662f\u6574\u6279\u6570\u636e\u7684\u51fd\u6570\u3002\u6211\u4eec\u5b9e\u9645\u8bc4\u4f30\u7684\u662f\u4e00\u4e2a\u5c0f\u6279\u91cf\uff0c\u6240\u4ee5\u5b9e\u9645\u7684\u529f\u80fd\u662f\u968f\u673a\u6307\u6807\u7684\u5b9e\u73b0_^_4_^_\u3002\u8fd9\u5c31\u662f\u6211\u4eec\u8c08\u8bba\u9884\u671f\u503c\u7684\u539f\u56e0\u3002\u56e0\u6b64\uff0c\u8ba9\u51fd\u6570\u5b9e\u73b0_^_5_^_\u9002\u7528\u4e8e\u8bad\u7ec3\u7684\u6bcf\u4e2a\u65f6\u95f4\u6b65\u3002
\n\u6211\u4eec\u5c06\u4f18\u5316\u5668\u7684\u6027\u80fd\u4f5c\u4e3a\u9057\u61be\u6765\u8861\u91cf\uff0c_^_6_^_\u5176\u4e2d_^_7_^_\u662f\u65f6\u95f4\u6b65\u7684\u53c2\u6570_^_8_^_\uff0c_^_9_^_\u662f\u6700\u4f73\u7684\u6700\u5c0f\u5316\u7684\u53c2\u6570_^_10_^_\u3002
\n\u73b0\u5728\u8ba9\u6211\u4eec\u6765\u5b9a\u4e49\u7efc\u5408\u95ee\u9898\uff0c
\n_^_11_^_\u5728\u54ea\u91cc_^_12_^_\u3002\u6700\u4f73\u7684\u89e3\u51b3\u65b9\u6848\u662f_^_13_^_\u3002
\n\u8fd9\u6bb5\u4ee3\u7801\u5c06\u5c1d\u8bd5\u8fd0\u884c\u4e9a\u5f53\u548c\u963f\u59c6\u65af\u683c\u62c9\u5fb7\u6765\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\u3002
\n", "_^_0_^_
\n": "_^_0_^_
\n", "Calculate _^_0_^_.
\n\ud83e\udd14 I feel you should be taking / maintaining the max of the bias corrected second exponential average of squared gradient. But this is how it's implemented in PyTorch also. I guess it doesn't really matter since bias correction only increases the value and it only makes an actual difference during the early few steps of the training.
\n": "\u8ba1\u7b97_^_0_^_\u3002
\n\ud83e\udd14 \u6211\u89c9\u5f97\u4f60\u5e94\u8be5\u53d6/\u4fdd\u6301\u504f\u5dee\u6821\u6b63\u7684\u5e73\u65b9\u68af\u5ea6\u7684\u7b2c\u4e8c\u4e2a\u6307\u6570\u5e73\u5747\u503c\u7684\u6700\u5927\u503c\u3002\u4f46\u8fd9\u4e5f\u662f\u5728 PyTorch \u4e2d\u5b9e\u73b0\u5b83\u7684\u65b9\u5f0f\u3002\u6211\u60f3\u8fd9\u5e76\u4e0d\u91cd\u8981\uff0c\u56e0\u4e3a\u504f\u5dee\u6821\u6b63\u53ea\u4f1a\u589e\u52a0\u503c\uff0c\u800c\u4e14\u53ea\u4f1a\u5728\u8bad\u7ec3\u7684\u6700\u521d\u51e0\u4e2a\u6b65\u9aa4\u4e2d\u4ea7\u751f\u5b9e\u9645\u5dee\u5f02\u3002
\n", "Calculate gradients
\n": "\u8ba1\u7b97\u68af\u5ea6
\n", "Call _^_0_^_ of Adam optimizer which we are extending
\n": "\u6211\u4eec\u6b63\u5728\u6269\u5c55_^_0_^_\u7684 Call of Adam \u4f18\u5316\u5668
\n", "Clear gradients
\n": "\u6e10\u53d8\u6e05\u6670
\n", "Create experiment to record results
\n": "\u521b\u5efa\u5b9e\u9a8c\u4ee5\u8bb0\u5f55\u7ed3\u679c
\n", "Define _^_0_^_ parameter
\n": "\u5b9a\u4e49_^_0_^_\u53c2\u6570
\n", "Fall back to Adam if the parameter group is not using _^_0_^_
\n": "\u5982\u679c\u53c2\u6570\u7ec4\u672a\u4f7f\u7528\uff0c\u5219\u56de\u9000\u5230 Adam_^_0_^_
\n", "Get _^_0_^_ and _^_1_^_ from Adam
\n": "_^_1_^_\u4ece Adam \u90a3\u91cc\u5f97_^_0_^_\u5230
\n", "Get _^_0_^_.
\n\ud83d\uddd2 The paper uses the notation _^_1_^_ for this, which we don't use that here because it confuses with the Adam's usage of the same notation for bias corrected exponential moving average.
\n": "\u5f97\u5230_^_0_^_\u3002
\n\ud83d\uddd2 \u672c\u6587\u4f7f\u7528\u4e86\u8fd9\u4e2a\u7b26\u53f7_^_1_^_\uff0c\u6211\u4eec\u5728\u8fd9\u91cc\u4e0d\u4f7f\u7528\u8fd9\u79cd\u7b26\u53f7\uff0c\u56e0\u4e3a\u5b83\u4e0e\u4e9a\u5f53\u5bf9\u504f\u5dee\u6821\u6b63\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u4f7f\u7528\u76f8\u540c\u7684\u7b26\u53f7\u6df7\u6dc6\u4e86\u3002
\n", "If _^_0_^_ flag is _^_1_^_ for this parameter group, we maintain the maximum of exponential moving average of squared gradient
\n": "\u5982\u679c f_^_0_^_ lag_^_1_^_ \u7528\u4e8e\u6b64\u53c2\u6570\u7ec4\uff0c\u5219\u6211\u4eec\u4fdd\u6301\u68af\u5ea6\u5e73\u65b9\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u7684\u6700\u5927\u503c
\n", "If this parameter group is using _^_0_^_
\n": "\u5982\u679c\u6b64\u53c2\u6570\u7ec4\u6b63\u5728\u4f7f\u7528_^_0_^_
\n", "Initialize the relevant optimizer
\n": "\u521d\u59cb\u5316\u76f8\u5173\u7684\u4f18\u5316\u5668
\n", "Make sure _^_0_^_
\n": "\u8bf7\u786e\u4fdd_^_0_^_
\n", "Optimal, _^_0_^_
\n": "\u6700\u4f73\uff0c_^_0_^_
\n", "Optimize
\n": "\u4f18\u5316
\n", "Run for _^_0_^_ steps
\n": "\u8dd1\u6b65\u8dd1_^_0_^_\u6b65
\n", "Run the synthetic experiment is AMSGrad You can see that AMSGrad converges to true optimal _^_0_^_
\n": "\u5728 amsGrad \u8fd0\u884c\u5408\u6210\u5b9e\u9a8c\u4f60\u53ef\u4ee5\u770b\u5230 amsGrad \u4f1a\u805a\u5230\u771f\u6b63\u7684\u6700\u4f18\u503c_^_0_^_
\n", "Run the synthetic experiment is Adam. You can see that Adam converges at _^_0_^_
\n": "\u8fd0\u884c\u5408\u6210\u5b9e\u9a8c\u7684\u662f\u4e9a\u5f53\u3002\u4f60\u53ef\u4ee5\u770b\u5230\u4e9a\u5f53\u805a\u96c6\u5728_^_0_^_
\n", "Track results every 1,000 steps
\n": "\u6bcf 1000 \u6b65\u8ddf\u8e2a\u4e00\u6b21\u7ed3\u679c
\n", "A simple PyTorch implementation/tutorial of AMSGrad optimizer.": "\u4e00\u4e2a\u7b80\u5355\u7684 AmsGrad \u4f18\u5316\u5668\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002", "AMSGrad Optimizer": "amsGrad \u4f18\u5316\u5668" }