This is a PyTorch implementation of popular optimizer Adam from paper Adam: A Method for Stochastic Optimization.
\nAdam update is,
\n_^_0_^_where _^_1_^_, _^_2_^_, _^_3_^_ and _^_4_^_ are scalar hyper parameters. _^_5_^_ and _^_6_^_ are first and second order moments. _^_7_^_ and _^_8_^_ are biased corrected moments. _^_9_^_ is used as a fix for division by zero error, but also acts as a form of a hyper-parameter that acts against variance in gradients.
\nEffective step taken assuming _^_10_^_ is, _^_11_^_ This is bounded by, _^_12_^_ when _^_13_^_ and _^_14_^_ otherwise. And in most common scenarios, _^_15_^_
\n": "\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c\u30a2\u30c0\u30e0\uff1a\u78ba\u7387\u7684\u6700\u9069\u5316\u306e\u65b9\u6cd5\u300d\u306b\u63b2\u8f09\u3055\u308c\u305f\u4eba\u6c17\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fcAdam\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059\u3002
\n\u30a2\u30c0\u30e0\u306e\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u306f\u3001
\n_^_0_^_\u3053\u3053\u3067_^_1_^_\u3001_^_2_^_\u3001_^_3_^__^_4_^_\u304a\u3088\u3073\u306f\u30b9\u30ab\u30e9\u30fc\u306e\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3067\u3059\u3002_^_5_^_\u30d5\u30a1\u30fc\u30b9\u30c8\u30aa\u30fc\u30c0\u30fc\u3001\u30bb\u30ab\u30f3\u30c9\u30aa\u30fc\u30c0\u30fc\u306e\u77ac\u9593\u3067\u3059 _^_6_^__^_7_^__^_8_^_\u504f\u308a\u4fee\u6b63\u3055\u308c\u305f\u30e2\u30fc\u30e1\u30f3\u30c8\u3067\u3059\u3002_^_9_^_\u30bc\u30ed\u30a8\u30e9\u30fc\u306b\u3088\u308b\u9664\u7b97\u306e\u4fee\u6b63\u3068\u3057\u3066\u4f7f\u308f\u308c\u307e\u3059\u304c\u3001\u52fe\u914d\u306e\u3070\u3089\u3064\u304d\u306b\u5bfe\u3057\u3066\u4f5c\u7528\u3059\u308b\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u5f62\u5f0f\u3068\u3057\u3066\u3082\u6a5f\u80fd\u3057\u307e\u3059
\u3002\n_^_10_^_\u6709\u52b9\u306a\u624b\u9806\u306f\u3001\u300c_^_11_^_This \u304c\u5236\u9650\u3055\u308c\u308b\u300d\u3001\u300c\u3044\u3064\u300d\u3001\u300c_^_12_^_\u305d\u308c\u4ee5\u5916\u306e\u5834\u5408_^_13_^_\u300d\u3092\u524d\u63d0\u3068\u3057\u3066\u3044\u307e\u3059\u3002_^_14_^_\u305d\u3057\u3066\u3001\u6700\u3082\u4e00\u822c\u7684\u306a\u30b7\u30ca\u30ea\u30aa\u3067\u306f\u3001_^_15_^_
\n", "We extend the class _^_0_^_ defined in _^_1_^_ to implement the Adam optimizer.
\n": "_^_0_^__^_1_^_\u3067\u5b9a\u7fa9\u3057\u305f\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3057\u3066 Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u5b9f\u88c5\u3057\u307e\u3059\u3002
\n", "This computes the following
\n_^_8_^_Since _^_9_^_, _^_10_^_, _^_11_^_ and _^_12_^_ are scalars and others are tensors we modify this calculation to optimize the computation.
\n_^_13_^_where _^_14_^_ is what we should specify as the hyper-parameter.
\n": "\u3053\u308c\u306b\u3088\u308a\u3001\u4ee5\u4e0b\u304c\u8a08\u7b97\u3055\u308c\u307e\u3059
\n_^_8_^__^_9_^__^_10_^_\u3001_^_11_^__^_12_^_\u306f\u30b9\u30ab\u30e9\u30fc\u3067\u3001\u305d\u306e\u4ed6\u306f\u30c6\u30f3\u30bd\u30eb\u306a\u306e\u3067\u3001\u3053\u306e\u8a08\u7b97\u3092\u5909\u66f4\u3057\u3066\u8a08\u7b97\u3092\u6700\u9069\u5316\u3057\u307e\u3059\u3002
\n_^_13_^__^_14_^_\u3053\u3053\u3067\u3001\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u3068\u3057\u3066\u6307\u5b9a\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002
\n", "This returns the modified learning rate based on the state. For Adam this is just the specified learning rate for the parameter group, _^_0_^_.
\n": "\u3053\u308c\u306b\u3088\u308a\u3001\u72b6\u614b\u306b\u57fa\u3065\u3044\u3066\u4fee\u6b63\u3055\u308c\u305f\u5b66\u7fd2\u7387\u304c\u8fd4\u3055\u308c\u307e\u3059\u3002Adam \u306e\u5834\u5408\u3001\u3053\u308c\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306b\u6307\u5b9a\u3055\u308c\u3066\u3044\u308b\u5b66\u7fd2\u7387\u306b\u3059\u304e\u307e\u305b\u3093_^_0_^_\u3002
\n", "_^_0_^_
\n": "_^_0_^_
\n", "Bias correction term for _^_0_^_, _^_1_^_
\n": "_^_0_^_\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u7528\u8a9e _^_1_^_
\n", "Calculate weight decay
\n": "\u4f53\u91cd\u6e1b\u5c11\u306e\u8a08\u7b97
\n", "Computation without optimization
\n": "\u6700\u9069\u5316\u306a\u3057\u306e\u8a08\u7b97
\n", "Exponential moving average of gradients, _^_0_^_
\n": "\u52fe\u914d\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3001_^_0_^_
\n", "Exponential moving average of squared gradient values, _^_0_^_
\n": "\u4e8c\u4e57\u52fe\u914d\u5024\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3001_^_0_^_
\n", "Get _^_0_^_ and _^_1_^_
\n": "_^_0_^_\u53d6\u5f97\u3057\u3066 _^_1_^_
\n", "Get learning rate
\n": "\u5b66\u7fd2\u7387\u3092\u53d6\u5f97
\n", "In-place calculation of _^_0_^_ _^_1_^_
\n": "\u306e\u30a4\u30f3\u30d7\u30ec\u30fc\u30b9\u8a08\u7b97 _^_0_^_ _^_1_^_
\n", "Increment _^_0_^_ the number of optimizer steps
\n": "_^_0_^_\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u30b9\u30c6\u30c3\u30d7\u6570\u3092\u5897\u3084\u3059
\n", "Perform Adam update
\n": "Adam \u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u3092\u5b9f\u884c
\n", "This is the number of optimizer steps taken on the parameter, _^_0_^_
\n": "\u3053\u308c\u306f\u3001\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u306b\u5bfe\u3057\u3066\u5b9f\u884c\u3055\u308c\u305f\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u30b9\u30c6\u30c3\u30d7\u306e\u6570\u3067\u3059\u3002_^_0_^_
\n", "Whether to optimize the computation
\n": "\u8a08\u7b97\u3092\u6700\u9069\u5316\u3059\u308b\u304b\u3069\u3046\u304b
\n", "A simple PyTorch implementation/tutorial of Adam optimizer": "Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u7c21\u5358\u306a PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb", "Adam Optimizer": "\u30a2\u30c0\u30e0\u30fb\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc" }