This is a PyTorch implementation of popular optimizer Adam from paper Adam: A Method for Stochastic Optimization.
\nAdam update is,
\n_^_0_^_where _^_1_^_, _^_2_^_, _^_3_^_ and _^_4_^_ are scalar hyper parameters. _^_5_^_ and _^_6_^_ are first and second order moments. _^_7_^_ and _^_8_^_ are biased corrected moments. _^_9_^_ is used as a fix for division by zero error, but also acts as a form of a hyper-parameter that acts against variance in gradients.
\nEffective step taken assuming _^_10_^_ is, _^_11_^_ This is bounded by, _^_12_^_ when _^_13_^_ and _^_14_^_ otherwise. And in most common scenarios, _^_15_^_
\n": "\u8fd9\u662f\u8bba\u6587\u300a\u4e9a\u5f53\uff1a\u968f\u673a\u4f18\u5316\u65b9\u6cd5\u300b\u4e2d\u6d41\u884c\u7684\u4f18\u5316\u5668 Adam \u7684 Py Torch \u5b9e\u73b0\u3002
\n\u4e9a\u5f53\u7684\u66f4\u65b0\u662f\uff0c
\n_^_0_^_\u5176\u4e2d_^_1_^__^_2_^_\u3001_^_3_^_\u548c_^_4_^_\u662f\u6807\u91cf\u8d85\u7ea7\u53c2\u6570\u3002_^_5_^_\u548c_^_6_^_\u662f\u4e00\u9636\u548c\u4e8c\u9636\u65f6\u523b\u3002_^_7_^_\u5e76\u4e14_^_8_^_\u662f\u6709\u504f\u5dee\u7684\u6821\u6b63\u65f6\u523b\u3002_^_9_^_\u7528\u4f5c\u9664\u4ee5\u96f6\u8bef\u5dee\u7684\u4fee\u590d\uff0c\u4f46\u4e5f\u7528\u4f5c\u5bf9\u68af\u5ea6\u65b9\u5dee\u8d77\u4f5c\u7528\u7684\u8d85\u53c2\u6570\u7684\u4e00\u79cd\u5f62\u5f0f\u3002
\n\u5047\u8bbe\u91c7\u53d6\u7684\u6709\u6548\u6b65\u9aa4_^_10_^_\u662f\uff0c_^_11_^_\u8fd9\u53d7\u9650\u4e8e\u3001_^_12_^_\u4f55\u65f6_^_13_^_\u4ee5\u53ca_^_14_^_\u5176\u4ed6\u65b9\u9762\u3002\u5728\u5927\u591a\u6570\u5e38\u89c1\u60c5\u51b5\u4e0b\uff0c_^_15_^_
\n", "We extend the class _^_0_^_ defined in _^_1_^_ to implement the Adam optimizer.
\n": "\u6211\u4eec\u6269\u5c55\u4e86\u4e2d_^_0_^_\u5b9a\u4e49\u7684\u7c7b_^_1_^_\u6765\u5b9e\u73b0 Adam \u4f18\u5316\u5668\u3002
\n", "This computes the following
\n_^_8_^_Since _^_9_^_, _^_10_^_, _^_11_^_ and _^_12_^_ are scalars and others are tensors we modify this calculation to optimize the computation.
\n_^_13_^_where _^_14_^_ is what we should specify as the hyper-parameter.
\n": "\u8fd9\u8ba1\u7b97\u51fa\u4ee5\u4e0b\u5185\u5bb9
\n_^_8_^_\u7531\u4e8e_^_9_^__^_10_^_\u3001_^_11_^_\u548c_^_12_^_\u662f\u6807\u91cf\uff0c\u5176\u4ed6\u662f\u5f20\u91cf\uff0c\u56e0\u6b64\u6211\u4eec\u5c06\u6b64\u8ba1\u7b97\u4fee\u6539\u4e3a\u4f18\u5316\u8ba1\u7b97\u3002
\n_^_13_^_wher_^_14_^_ e \u662f\u6211\u4eec\u5e94\u8be5\u6307\u5b9a\u4e3a\u8d85\u53c2\u6570\u7684\u5185\u5bb9\u3002
\n", "This returns the modified learning rate based on the state. For Adam this is just the specified learning rate for the parameter group, _^_0_^_.
\n": "\u8fd9\u5c06\u6839\u636e\u72b6\u6001\u8fd4\u56de\u4fee\u6539\u540e\u7684\u5b66\u4e60\u901f\u7387\u3002\u5bf9\u4e8e Adam \u6765\u8bf4\uff0c\u8fd9\u53ea\u662f\u53c2\u6570\u7ec4\u7684\u6307\u5b9a\u5b66\u4e60\u901f\u7387_^_0_^_\u3002
\n", "_^_0_^_
\n": "_^_0_^_
\n", "Bias correction term for _^_0_^_, _^_1_^_
\n": "\u504f\u5dee\u6821\u6b63\u672f\u8bed_^_0_^_\uff0c_^_1_^_
\n", "Calculate weight decay
\n": "\u8ba1\u7b97\u4f53\u91cd\u8870\u51cf
\n", "Computation without optimization
\n": "\u65e0\u9700\u4f18\u5316\u7684\u8ba1\u7b97
\n", "Exponential moving average of gradients, _^_0_^_
\n": "\u68af\u5ea6\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c_^_0_^_
\n", "Exponential moving average of squared gradient values, _^_0_^_
\n": "\u68af\u5ea6\u5e73\u65b9\u503c\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c_^_0_^_
\n", "Get _^_0_^_ and _^_1_^_
\n": "\u83b7\u53d6_^_0_^_\u548c_^_1_^_
\n", "Get learning rate
\n": "\u83b7\u53d6\u5b66\u4e60\u7387
\n", "In-place calculation of _^_0_^_ _^_1_^_
\n": "\u5c31\u5730\u8ba1\u7b97_^_0_^__^_1_^_
\n", "Increment _^_0_^_ the number of optimizer steps
\n": "_^_0_^_\u589e\u52a0\u4f18\u5316\u5668\u6b65\u6570
\n", "Perform Adam update
\n": "\u6267\u884c Adam \u66f4\u65b0
\n", "This is the number of optimizer steps taken on the parameter, _^_0_^_
\n": "\u8fd9\u662f\u4f18\u5316\u5668\u5bf9\u53c2\u6570\u91c7\u53d6\u7684\u6b65\u9aa4\u6570\uff0c_^_0_^_
\n", "Whether to optimize the computation
\n": "\u662f\u5426\u4f18\u5316\u8ba1\u7b97
\n", "A simple PyTorch implementation/tutorial of Adam optimizer": "Adam \u4f18\u5316\u5668\u7684\u4e00\u4e2a\u7b80\u5355\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b", "Adam Optimizer": "\u4e9a\u5f53\u4f18\u5316\u5668" }