We extend Adam Optimizer but use FP32 to store gradients and moments.
\n": "\u6211\u4eec\u6269\u5c55\u4e86 Adam Optimizer\uff0c\u4f46\u4f7f\u7528 FP32 \u6765\u5b58\u50a8\u6e10\u53d8\u548c\u65f6\u523b\u3002
\n", "We extend PyTorch gradient scaler to use FP32 gradients.
\n": "\u6211\u4eec\u5c06 PyTorch \u68af\u5ea6\u7f29\u653e\u5668\u6269\u5c55\u4e3a\u4f7f\u7528 FP32 \u6e10\u53d8\u3002
\n", "All the state tensors use FP32.
\n": "\u6240\u6709\u72b6\u6001\u5f20\u91cf\u90fd\u4f7f\u7528 FP32\u3002
\n", "\n": "\n", "
Calculate weight decay
\n": "\u8ba1\u7b97\u4f53\u91cd\u8870\u51cf
\n", "Call the Adam Optimizer initializer
\n": "\u8c03\u7528 Adam \u4f18\u5316\u5668\u521d\u59cb\u5316\u5668
\n", "Exponential moving average of gradients, _^_0_^_
\n": "\u68af\u5ea6\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c_^_0_^_
\n", "Exponential moving average of squared gradient values, _^_0_^_
\n": "\u68af\u5ea6\u5e73\u65b9\u503c\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c_^_0_^_
\n", "Get _^_0_^_ and _^_1_^_
\n": "\u83b7\u53d6_^_0_^_\u548c_^_1_^_
\n", "Get the FP32 gradients if available
\n": "\u83b7\u53d6 FP32 \u6e10\u53d8\uff08\u5982\u679c\u6709\uff09
\n", "Get the FP32 parameters
\n": "\u83b7\u53d6 FP32 \u53c2\u6570
\n", "If we are using the _^_0_^_ optimizer set _^_1_^_ to the FP32 gradients
\n": "\u5982\u679c\u6211\u4eec\u4f7f\u7528\u8bbe\u7f6e\u4e3a_^_1_^_ FP32 \u6e10\u53d8\u7684_^_0_^_\u4f18\u5316\u5668
\n", "Increment _^_0_^_ the number of optimizer steps
\n": "_^_0_^_\u589e\u52a0\u4f18\u5316\u5668\u6b65\u6570
\n", "Loop through parameters
\n": "\u5faa\u73af\u6d4f\u89c8\u53c2\u6570
\n", "Maintain a FP32 copy of the parameters
\n": "\u7ef4\u62a4\u53c2\u6570\u7684 FP32 \u526f\u672c
\n", "Not implemented for sparse tensors
\n": "\u672a\u9488\u5bf9\u7a00\u758f\u5f20\u91cf\u5b9e\u73b0
\n", "Otherwise, convert the gradients to FP32
\n": "\u5426\u5219\uff0c\u5c06\u6e10\u53d8\u8f6c\u6362\u4e3a FP32
\n", "Otherwise, do not convert the gradients to FP32
\n": "\u5426\u5219\uff0c\u4e0d\u8981\u5c06\u6e10\u53d8\u8f6c\u6362\u4e3a FP32
\n", "Parameter to store 32 bit gradients. This get populated by the _^_0_^_ defined below.
\n": "\u7528\u4e8e\u5b58\u50a8 32 \u4f4d\u6e10\u53d8\u7684\u53c2\u6570\u3002\u8fd9\u7531\u4e0b\u9762_^_0_^_\u5b9a\u4e49\u7684\u586b\u5145\u3002
\n", "Perform Adam update
\n": "\u6267\u884c Adam \u66f4\u65b0
\n", "Set the parameters
\n": "\u8bbe\u7f6e\u53c2\u6570
\n", "Skip non-trainable parameters
\n": "\u8df3\u8fc7\u4e0d\u53ef\u8bad\u7ec3\u7684\u53c2\u6570
\n", "This is the number of optimizer steps taken on the parameter, _^_0_^_
\n": "\u8fd9\u662f\u4f18\u5316\u5668\u5bf9\u53c2\u6570\u91c7\u53d6\u7684\u6b65\u9aa4\u6570\uff0c_^_0_^_
\n", "Unscale all the gradients
\n": "\u53d6\u6d88\u7f29\u653e\u6240\u6709\u6e10\u53d8
\n", "A simple PyTorch implementation/tutorial of Adam optimizer": "Adam \u4f18\u5316\u5668\u7684\u4e00\u4e2a\u7b80\u5355\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b", "Adam Optimizer for Half Precision Training": "\u534a\u7cbe\u5ea6\u8bad\u7ec3\u7684 Adam Optimizer" }