chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"<h1>Optimizers</h1>\n<h2>Optimizer Implementations</h2>\n<ul><li><a href=\"adam.html\">Adam Optimizer</a> </li>\n<li><a href=\"amsgrad.html\">AMSGrad Optimizer</a> </li>\n<li><a href=\"adam_warmup.html\">Adam Optimizer with warmup</a> </li>\n<li><a href=\"noam.html\">Noam Optimizer</a> </li>\n<li><a href=\"radam.html\">Rectified Adam Optimizer</a> </li>\n<li><a href=\"ada_belief.html\">AdaBelief Optimizer</a> </li>\n<li><a href=\"sophia.html\">Sophia-G Optimizer</a></li></ul>\n<p>This <a href=\"mnist_experiment.html\">MNIST example</a> uses these optimizers.</p>\n<h2>Generic Adaptive Optimizer Base class and Weight Decay</h2>\n<p>This file defines a common base class for <em>Adam</em> and extensions of it. The base class helps use implement other optimizers with minimal code because of re-usability.</p>\n<p>We also define a special class for L2 weight decay, so that we don't have to implement it inside each of the optimizers, and can easily extend to other weight decays like L1 without changing the optimizers.</p>\n<p>Here are some concepts on PyTorch optimizers:</p>\n<h3>Parameter groups</h3>\n<p>PyTorch optimizers group parameters into sets called groups. Each group can have its own hyper-parameters like learning rates.</p>\n<p>In most common cases there will be only one group. This is when you initialize your optimizer with,</p>\n<span translate=no>_^_0_^_</span><p>You can define multiple parameter groups when initializing the optimizer:</p>\n<span translate=no>_^_1_^_</span><p>Here we pass a list of groups. Each group is a dictionary with its parameters under the key 'params'. You specify any hyper-parameters as well. If the hyper parameters are not defined they will default to the optimizer level defaults.</p>\n<p>You can access (and even change) these groups, and their hyper-parameters with <span translate=no>_^_2_^_</span>. Most learning rate schedule implementations I've come across do access this and change 'lr'.</p>\n<h3>States</h3>\n<p>Optimizer maintains states (a dictionary) for each parameter (a tensor), in a dictionary <span translate=no>_^_3_^_</span>. This is where the optimizer maintains things like exponential averages.</p>\n": "<h1>Optimizers</h1>\n<h2>Optimizer Implementations</h2>\n<ul><li><a href=\"adam.html\">Adam Optimizer</a> </li>\n<li><a href=\"amsgrad.html\">AMSGrad Optimizer</a> </li>\n<li><a href=\"adam_warmup.html\">Adam Optimizer with warmup</a> </li>\n<li><a href=\"noam.html\">Noam Optimizer</a> </li>\n<li><a href=\"radam.html\">Rectified Adam Optimizer</a> </li>\n<li><a href=\"ada_belief.html\">AdaBelief Optimizer</a> </li>\n<li><a href=\"sophia.html\">Sophia-G Optimizer</a></li></ul>\n<p>This <a href=\"mnist_experiment.html\">MNIST example</a> uses these optimizers.</p>\n<h2>Generic Adaptive Optimizer Base class and Weight Decay</h2>\n<p>This file defines a common base class for <em>Adam</em> and extensions of it. The base class helps use implement other optimizers with minimal code because of re-usability.</p>\n<p>We also define a special class for L2 weight decay, so that we don't have to implement it inside each of the optimizers, and can easily extend to other weight decays like L1 without changing the optimizers.</p>\n<p>Here are some concepts on PyTorch optimizers:</p>\n<h3>Parameter groups</h3>\n<p>PyTorch optimizers group parameters into sets called groups. Each group can have its own hyper-parameters like learning rates.</p>\n<p>In most common cases there will be only one group. This is when you initialize your optimizer with,</p>\n<span translate=no>_^_0_^_</span><p>You can define multiple parameter groups when initializing the optimizer:</p>\n<span translate=no>_^_1_^_</span><p>Here we pass a list of groups. Each group is a dictionary with its parameters under the key 'params'. You specify any hyper-parameters as well. If the hyper parameters are not defined they will default to the optimizer level defaults.</p>\n<p>You can access (and even change) these groups, and their hyper-parameters with <span translate=no>_^_2_^_</span>. Most learning rate schedule implementations I've come across do access this and change 'lr'.</p>\n<h3>States</h3>\n<p>Optimizer maintains states (a dictionary) for each parameter (a tensor), in a dictionary <span translate=no>_^_3_^_</span>. This is where the optimizer maintains things like exponential averages.</p>\n",
|
||||
"<h2>Base class for <em>Adam</em> and extensions</h2>\n": "<h2><em>Adam</em> \u548c\u6269\u5c55\u7684\u57fa\u7c7b</h2>\n",
|
||||
"<h2>L2 Weight decay</h2>\n": "<h2>L2 \u91cd\u91cf\u8870\u51cf</h2>\n",
|
||||
"<h3>Initialize state for a given parameter tensor</h3>\n<p>This should be overridden with code to initialize <span translate=no>_^_0_^_</span> for parameters <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> is the parameter group dictionary to which <span translate=no>_^_3_^_</span> belongs.</p>\n": "<h3>\u521d\u59cb\u5316\u7ed9\u5b9a\u53c2\u6570\u5f20\u91cf\u7684\u72b6\u6001</h3>\n<p>\u8fd9\u5e94\u8be5\u88ab\u4ee3\u7801\u8986\u76d6\uff0c\u4ee5\u4fbf\u521d\u59cb<span translate=no>_^_0_^_</span>\u5316\u53c2\u6570<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u662f\u6240<span translate=no>_^_3_^_</span>\u5c5e\u7684\u53c2\u6570\u7ec4\u5b57\u5178\u3002</p>\n",
|
||||
"<h3>Initialize weight decay</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the decay coefficient </li>\n<li><span translate=no>_^_1_^_</span> is a flag indicating whether to add the weight decay to the gradient or directly decay from the parameter. If added to the gradient it will go through the normal optimizer update. </li>\n<li><span translate=no>_^_2_^_</span> this flag indicates whether the weight decay coefficient is absolute. This is applicable when the decay is performed directly on the parameter. If this is false the actual decay is <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span>.</li></ul>\n": "<h3>\u521d\u59cb\u5316\u6743\u91cd\u8870\u51cf</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u8870\u51cf\u7cfb\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u6307\u793a\u662f\u5c06\u6743\u91cd\u8870\u51cf\u6dfb\u52a0\u5230\u68af\u5ea6\u8fd8\u662f\u76f4\u63a5\u4ece\u53c2\u6570\u4e2d\u8870\u51cf\u3002\u5982\u679c\u6dfb\u52a0\u5230\u6e10\u53d8\u4e2d\uff0c\u5b83\u5c06\u901a\u8fc7\u666e\u901a\u7684\u4f18\u5316\u5668\u66f4\u65b0\u3002</li>\n<li><span translate=no>_^_2_^_</span>\u6b64\u6807\u5fd7\u6307\u793a\u6743\u91cd\u8870\u51cf\u7cfb\u6570\u662f\u5426\u4e3a\u7edd\u5bf9\u503c\u3002\u5f53\u76f4\u63a5\u5bf9\u53c2\u6570\u6267\u884c\u8870\u51cf\u65f6\uff0c\u8fd9\u9002\u7528\u3002\u5982\u679c\u6b64\u503c\u4e3a\u5047\uff0c\u5219\u5b9e\u9645\u8870\u51cf\u4e3a<span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Initialize</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the collection of parameters or set of parameter groups. </li>\n<li><span translate=no>_^_1_^_</span> a dictionary of default hyper-parameters </li>\n<li><span translate=no>_^_2_^_</span> is the learning rate, <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> is the tuple <span translate=no>_^_5_^_</span> </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span></li></ul>\n": "<h3>\u521d\u59cb\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\u7684\u96c6\u5408\u6216\u4e00\u7ec4\u53c2\u6570\u7ec4\u3002</li>\n<li><span translate=no>_^_1_^_</span>\u9ed8\u8ba4\u8d85\u53c2\u6570\u7684\u5b57\u5178</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u5b66\u4e60\u7387\uff0c<span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u662f\u5143\u7ec4<span translate=no>_^_5_^_</span></li>\n</ul><li><span translate=no>_^_6_^_</span>\u662f<span translate=no>_^_7_^_</span></li>\n",
|
||||
"<h3>Optimizer step</h3>\n<p>We have created a template method that does the common stuff every <em>Adam</em> based optimizer needs.</p>\n": "<h3>\u4f18\u5316\u5668\u6b65\u9aa4</h3>\n<p>\u6211\u4eec\u521b\u5efa\u4e86\u4e00\u4e2a\u6a21\u677f\u65b9\u6cd5\uff0c\u5b83\u53ef\u4ee5\u5b8c\u6210\u6bcf\u4e2a\u57fa\u4e8e <em>Adam</em> \u7684\u4f18\u5316\u5668\u6240\u9700\u8981\u7684\u5e38\u7528\u5185\u5bb9\u3002</p>\n",
|
||||
"<h3>Perform weight decay and return the gradient</h3>\n": "<h3>\u6267\u884c\u6743\u91cd\u8870\u51cf\u5e76\u8fd4\u56de\u68af\u5ea6</h3>\n",
|
||||
"<h3>Take optimizer step on a parameter tensor</h3>\n<p>This should be overridden and take the optimization step on <span translate=no>_^_0_^_</span> tensor <span translate=no>_^_1_^_</span>, where <span translate=no>_^_2_^_</span> is the gradient for that parameter, <span translate=no>_^_3_^_</span>, <span translate=no>_^_4_^_</span> is the optimizer state dictionary for that parameter, and <span translate=no>_^_5_^_</span> is the parameter group dictionary <span translate=no>_^_6_^_</span> belongs to.</p>\n": "<h3>\u5728\u53c2\u6570\u5f20\u91cf\u4e0a\u91c7\u53d6\u4f18\u5316\u5668\u6b65\u9aa4</h3>\n<p>\u8fd9\u5e94\u8be5\u88ab\u91cd\u5199\u5e76\u5bf9<span translate=no>_^_0_^_</span>\u5f20\u91cf\u91c7\u53d6\u4f18\u5316\u6b65\u9aa4<span translate=no>_^_1_^_</span>\uff0c\u5176\u4e2d<span translate=no>_^_2_^_</span>\u662f\u8be5\u53c2\u6570\u7684\u68af\u5ea6<span translate=no>_^_3_^_</span>\uff0c<span translate=no>_^_4_^_</span>\u662f\u8be5\u53c2\u6570\u7684\u4f18\u5316\u5668\u72b6\u6001\u5b57\u5178\uff0c<span translate=no>_^_5_^_</span>\u4e5f\u662f\u53c2\u6570\u7ec4\u5b57\u5178<span translate=no>_^_6_^_</span>\u6240\u5c5e\u7684\u3002</p>\n",
|
||||
"<p> Return defaults for parameter groups</p>\n": "<p>\u8fd4\u56de\u53c2\u6570\u7ec4\u7684\u9ed8\u8ba4\u503c</p>\n",
|
||||
"<p>Add the hyper-parameters to the defaults </p>\n": "<p>\u5c06\u8d85\u53c2\u6570\u6dfb\u52a0\u5230\u9ed8\u8ba4\u503c</p>\n",
|
||||
"<p>Add the weight decay to the gradient and return the modified gradient </p>\n": "<p>\u5c06\u6743\u91cd\u8870\u51cf\u6dfb\u52a0\u5230\u6e10\u53d8\u5e76\u8fd4\u56de\u4fee\u6539\u540e\u7684\u6e10\u53d8</p>\n",
|
||||
"<p>Calculate loss.</p>\n<p>\ud83e\udd14 I'm not sure when you need this. I guess it's if you define a function that calculates the loss, does <span translate=no>_^_0_^_</span> and return the loss, instead of calling it on your own you could pass it to <span translate=no>_^_1_^_</span>. \ud83e\udd37\u200d\u2642\ufe0f </p>\n": "<p>\u8ba1\u7b97\u635f\u5931\u3002</p>\n<p>\ud83e\udd14 \u6211\u4e0d\u786e\u5b9a\u4f60\u4ec0\u4e48\u65f6\u5019\u9700\u8981\u8fd9\u4e2a\u3002\u6211\u60f3\u5982\u679c\u4f60\u5b9a\u4e49\u4e00\u4e2a\u51fd\u6570\u6765\u8ba1\u7b97\u635f\u5931\uff0c\u505a<span translate=no>_^_0_^_</span>\u548c\u8fd4\u56de\u635f\u5931\uff0c\u800c\u4e0d\u662f\u81ea\u5df1\u8c03\u7528\u5b83\uff0c\u4f60\u53ef\u4ee5\u4f20\u9012\u7ed9\u5b83<span translate=no>_^_1_^_</span>\u3002\ud83e\udd37\u200d\u2642\ufe0f</p>\n",
|
||||
"<p>Check hyper-parameters </p>\n": "<p>\u68c0\u67e5\u8d85\u53c2\u6570</p>\n",
|
||||
"<p>Check the hyper-parameters </p>\n": "<p>\u68c0\u67e5\u8d85\u53c2\u6570</p>\n",
|
||||
"<p>Get the gradient tensor </p>\n": "<p>\u83b7\u53d6\u68af\u5ea6\u5f20\u91cf</p>\n",
|
||||
"<p>Get the state for the parameter </p>\n": "<p>\u83b7\u53d6\u53c2\u6570\u7684\u72b6\u6001</p>\n",
|
||||
"<p>If the weight decay coefficient is absolute </p>\n": "<p>\u5982\u679c\u6743\u91cd\u8870\u51cf\u7cfb\u6570\u4e3a\u7edd\u5bf9\u503c</p>\n",
|
||||
"<p>If we are doing the decay on the parameter directly </p>\n": "<p>\u5982\u679c\u6211\u4eec\u76f4\u63a5\u5bf9\u53c2\u6570\u8fdb\u884c\u8870\u51cf</p>\n",
|
||||
"<p>Initialize the PyTorch optimizer. This will create parameter groups with the default hyper-parameters </p>\n": "<p>\u521d\u59cb\u5316 PyTorch \u4f18\u5316\u5668\u3002\u8fd9\u5c06\u4f7f\u7528\u9ed8\u8ba4\u7684\u8d85\u53c2\u6570\u521b\u5efa\u53c2\u6570\u7ec4</p>\n",
|
||||
"<p>Initialize the state if state is uninitialized </p>\n": "<p>\u5982\u679c\u72b6\u6001\u672a\u521d\u59cb\u5316\uff0c\u5219\u521d\u59cb\u5316\u72b6\u6001</p>\n",
|
||||
"<p>Iterate through the parameter groups </p>\n": "<p>\u904d\u5386\u53c2\u6570\u7ec4</p>\n",
|
||||
"<p>Iterate through the parameters in the parameter group </p>\n": "<p>\u904d\u5386\u53c2\u6570\u7ec4\u4e2d\u7684\u53c2\u6570</p>\n",
|
||||
"<p>Otherwise, </p>\n": "<p>\u5426\u5219\uff0c</p>\n",
|
||||
"<p>Return the loss, calculated from closure </p>\n": "<p>\u8fd4\u56de\u4ece\u95ed\u5305\u8ba1\u7b97\u5f97\u51fa\u7684\u635f\u5931</p>\n",
|
||||
"<p>Return the unmodified gradient </p>\n": "<p>\u8fd4\u56de\u672a\u4fee\u6539\u7684\u6e10\u53d8</p>\n",
|
||||
"<p>Skip if the parameter has no gradient </p>\n": "<p>\u5982\u679c\u53c2\u6570\u6ca1\u6709\u6e10\u53d8\uff0c\u5219\u8df3\u8fc7</p>\n",
|
||||
"<p>Take the optimization step on the parameter </p>\n": "<p>\u5bf9\u53c2\u6570\u91c7\u53d6\u4f18\u5316\u6b65\u9aa4</p>\n",
|
||||
"<p>We don't handle sparse gradients </p>\n": "<p>\u6211\u4eec\u4e0d\u5904\u7406\u7a00\u758f\u6e10\u53d8</p>\n",
|
||||
"A set of PyTorch implementations/tutorials of popular gradient descent based optimizers. Currently includes Adam, AMSGrad and RAdam optimizers.": "\u4e00\u7ec4\u6d41\u884c\u7684\u57fa\u4e8e\u68af\u5ea6\u4e0b\u964d\u7684\u4f18\u5316\u5668\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002\u76ee\u524d\u5305\u62ec Adam\u3001AmsGrad \u548c RadAM \u4f18\u5316\u5668\u3002",
|
||||
"Optimizers": "\u4f18\u5316\u5668"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1>AdaBelief Optimizer</h1>\n<p>This is based from AdaBelief <a href=\"https://github.com/juntang-zhuang/Adabelief-Optimizer\">official implementation</a> of the paper <a href=\"https://arxiv.org/abs/2010.07468\">AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients</a>.</p>\n<p>This is implemented in <a href=\"https://pytorch.org\">PyTorch</a> as an extension to <a href=\"radam.html\">RAdam</a>.</p>\n<p>The main difference between Adam optimizer and AdaBelief is that, how it calculates the adaptive learning rate; instead of dividing by the exponential moving average of square of the gradients, AdaBelief divides by the exponential mean of variance.</p>\n<span translate=no>_^_0_^_</span><p>\ud83e\udd14 The paper calculates variance as <span translate=no>_^_1_^_</span>, but I feel it should use the bias corrected momentum <span translate=no>_^_2_^_</span>. I guess this doesn't affect things much because bias correction is <span translate=no>_^_3_^_</span> after the initial training steps.</p>\n": "<h1>\u30a2\u30c0\u30d6\u30ea\u30ea\u30fc\u30d5\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h1>\n<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/2010.07468\">Adabelief\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\uff1a\u89b3\u6e2c\u3055\u308c\u305f\u52fe\u914d\u3092\u4fe1\u3058\u3066\u30b9\u30c6\u30c3\u30d7\u30b5\u30a4\u30ba\u3092\u8abf\u6574\u3059\u308b\u300d<a href=\"https://github.com/juntang-zhuang/Adabelief-Optimizer\">\u3068\u3044\u3046\u8ad6\u6587\u306eAdableLief\u516c\u5f0f\u5b9f\u88c5\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a></a>\u3002</p>\n<p><a href=\"radam.html\">\u3053\u308c\u306f RadAM \u306e\u62e1\u5f35\u6a5f\u80fd\u3068\u3057\u3066 <a href=\"https://pytorch.org\">PyTorch</a> \u306b\u5b9f\u88c5\u3055\u308c\u3066\u3044\u307e\u3059\u3002</a></p>\n<p>Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3068 Adabelief \u306e\u4e3b\u306a\u9055\u3044\u306f\u3001\u9069\u5fdc\u578b\u5b66\u7fd2\u7387\u306e\u8a08\u7b97\u65b9\u6cd5\u306b\u3042\u308a\u307e\u3059\u3002Adabelief \u3067\u306f\u3001\u52fe\u914d\u306e 2 \u4e57\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3067\u5272\u308b\u306e\u3067\u306f\u306a\u304f\u3001\u6307\u6570\u95a2\u6570\u7684\u5206\u6563\u5e73\u5747\u3067\u9664\u7b97\u3055\u308c\u307e\u3059\u3002</p>\n<span translate=no>_^_0_^_</span><p>\ud83e\udd14 \u8ad6\u6587\u3067\u306f\u5206\u6563\u3092\u6b21\u306e\u3088\u3046\u306b\u8a08\u7b97\u3057\u3066\u3044\u307e\u3059\u304c<span translate=no>_^_1_^_</span>\u3001\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3055\u308c\u305f\u30e2\u30e1\u30f3\u30bf\u30e0\u3092\u4f7f\u7528\u3059\u3079\u304d\u3060\u3068\u601d\u3044\u307e\u3059\u3002<span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u306f\u6700\u521d\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b9\u30c6\u30c3\u30d7\u306e\u5f8c\u306b\u884c\u308f\u308c\u308b\u306e\u3067\u3001\u3053\u308c\u306f\u3042\u307e\u308a\u5f71\u97ff\u3057\u306a\u3044\u3068\u601d\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<h2>AdaBelief Optimizer</h2>\n<p>This class extends from RAdam optimizer defined in <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>\u30a2\u30c0\u30d6\u30ea\u30ea\u30fc\u30d5\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h2>\n<p>\u3053\u306e\u30af\u30e9\u30b9\u306f\u3001\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b RadAM \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u62e1\u5f35\u3057\u305f\u3082\u306e\u3067\u3059\u3002<a href=\"radam.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Calculate <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> or <span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_4_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_5_^_</span> is the current gradient tensor <span translate=no>_^_6_^_</span> for the parameter <span translate=no>_^_7_^_</span></li></ul>\n": "<h3><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u8a08\u7b97\u307e\u305f\u306f <span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_4_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_5_^_</span><span translate=no>_^_6_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u73fe\u5728\u306e\u52fe\u914d\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_7_^_</span></li></ul>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>\u30d1\u30e9\u30e1\u30fc\u30bf\u72b6\u614b\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_16_^_</span> whether to use sgd when the rectification term <span translate=no>_^_17_^_</span> is intractable </li>\n<li><span translate=no>_^_18_^_</span> is whether to use RAdam update </li>\n<li><span translate=no>_^_19_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_20_^_</span>.</li></ul>\n": "<h3>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u30ea\u30b9\u30c8\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u5b66\u7fd2\u7387 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>(,) <span translate=no>_^_4_^_</span> \u306e\u30bf\u30d7\u30eb\u3067\u3059 <span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span>\u307e\u305f\u306f\u305d\u308c\u306b\u57fa\u3065\u3044\u3066\u3044\u308b <span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span><span translate=no>_^_11_^_</span>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3067\u3059 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li><span translate=no>_^_13_^_</span>\u30bb\u30ab\u30f3\u30c9\u30e2\u30fc\u30e1\u30f3\u30c8\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3092\u52a0\u7b97\u3057\u3066\u304b\u3089\u884c\u3046\u3053\u3068\u3067\u6700\u9069\u5316\u3059\u308b\u304b\u5426\u304b\u306e\u30d5\u30e9\u30b0\u3067\u3059 <span translate=no>_^_14_^_</span></li>\n<li><span translate=no>_^_15_^_</span>amsGrad\u3092\u4f7f\u7528\u3059\u308b\u304b\u3001\u30d7\u30ec\u30fc\u30f3\u306aAdam\u306b\u30d5\u30a9\u30fc\u30eb\u30d0\u30c3\u30af\u3059\u308b\u304b\u3092\u793a\u3059\u30d5\u30e9\u30b0\u3067\u3059</li>\n<li><span translate=no>_^_16_^_</span>\u4fee\u6b63\u9805\u304c\u6271\u3044\u306b\u304f\u3044\u5834\u5408\u306b sgd \u3092\u4f7f\u3046\u304b\u3069\u3046\u304b <span translate=no>_^_17_^_</span></li>\n<li><span translate=no>_^_18_^_</span>RadAM\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u3092\u4f7f\u7528\u3059\u308b\u304b\u3069\u3046\u304b\u3067\u3059</li>\n<li><span translate=no>_^_19_^_</span>\u30b0\u30eb\u30fc\u30d7\u5024\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u8f9e\u66f8\u3067\u3059\u3002\u3053\u308c\u306f\u3001\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3059\u308b\u5834\u5408\u306b\u4fbf\u5229\u3067\u3059<span translate=no>_^_20_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u4e0e\u3048\u3089\u308c\u305f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb\u306e\u66f4\u65b0\u30b9\u30c6\u30c3\u30d7\u3092\u5b9f\u884c\u3059\u308b</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u73fe\u5728\u306e\u52fe\u914d\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> otherwise </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u305d\u308c\u4ee5\u5916\u306f</p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u8a08\u7b97<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u4f53\u91cd\u6e1b\u5c11\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Difference between gradient and momentum </p>\n": "<p>\u52fe\u914d\u3068\u904b\u52d5\u91cf\u306e\u9055\u3044</p>\n",
|
||||
"<p>Exponential moving average of gradient values </p>\n": "<p>\u52fe\u914d\u5024\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747</p>\n",
|
||||
"<p>Exponential moving average of variance </p>\n": "<p>\u6307\u6570\u79fb\u52d5\u5e73\u5747\u504f\u5dee</p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u53d6\u5f97\u3057\u3066 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u53d6\u5f97<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> flag is <span translate=no>_^_1_^_</span> for this parameter group, we maintain the maximum of exponential moving average of variance </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3053\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306b\u30d5\u30e9\u30b0\u3092\u6307\u5b9a\u3059\u308b\u3068\u3001\u6307\u6570\u79fb\u52d5\u5e73\u5747\u306e\u6700\u5927\u5206\u6563\u5024\u304c\u7dad\u6301\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p>If this parameter group is using <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3053\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u304c\u4f7f\u7528\u3057\u3066\u3044\u308b\u5834\u5408 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>In-place calculation of <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n": "<p>\u306e\u30a4\u30f3\u30d7\u30ec\u30fc\u30b9\u8a08\u7b97 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p><span translate=no>_^_0_^_</span>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u30b9\u30c6\u30c3\u30d7\u6570\u3092\u5897\u3084\u3059</p>\n",
|
||||
"<p>Maintains max of all exp. moving avg. of sq. grad. values </p>\n": "<p>\u3059\u3079\u3066\u306e\u8a31\u5bb9\u504f\u5dee\u79fb\u52d5\u5e73\u5747\u5024\u306e\u6700\u5927\u5024\u3092\u7dad\u6301</p>\n",
|
||||
"<p>Perform <em>Adam</em> update, defined in <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>, with <span translate=no>_^_1_^_</span> in place of <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u306e\u4ee3\u308f\u308a\u306b<a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>\u3001<span translate=no>_^_1_^_</span>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b <em>Adam</em> \u66f4\u65b0\u3092\u5b9f\u884c\u3057\u307e\u3059<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"<p>Perform <em>Rectified Adam</em> update defined in <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>, with <span translate=no>_^_1_^_</span> in place of <span translate=no>_^_2_^_</span>. </p>\n": "<p><em>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u4fee\u6b63\u6e08\u307f\u306e Adam</em> \u66f4\u65b0\u3092<a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>\u3001<span translate=no>_^_1_^_</span>\u306e\u4ee3\u308f\u308a\u306b\u3067\u5b9f\u884c\u3057\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n",
|
||||
"A simple PyTorch implementation/tutorial of AdaBelief optimizer.": "Adabelief \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u7c21\u5358\u306a PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002",
|
||||
"AdaBelief optimizer": "\u30a2\u30c0\u30d6\u30ea\u30ea\u30fc\u30d5\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1>AdaBelief Optimizer</h1>\n<p>This is based from AdaBelief <a href=\"https://github.com/juntang-zhuang/Adabelief-Optimizer\">official implementation</a> of the paper <a href=\"https://arxiv.org/abs/2010.07468\">AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients</a>.</p>\n<p>This is implemented in <a href=\"https://pytorch.org\">PyTorch</a> as an extension to <a href=\"radam.html\">RAdam</a>.</p>\n<p>The main difference between Adam optimizer and AdaBelief is that, how it calculates the adaptive learning rate; instead of dividing by the exponential moving average of square of the gradients, AdaBelief divides by the exponential mean of variance.</p>\n<span translate=no>_^_0_^_</span><p>\ud83e\udd14 The paper calculates variance as <span translate=no>_^_1_^_</span>, but I feel it should use the bias corrected momentum <span translate=no>_^_2_^_</span>. I guess this doesn't affect things much because bias correction is <span translate=no>_^_3_^_</span> after the initial training steps.</p>\n": "<h1>ADABelief\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h1>\n<p>\u0db8\u0dd9\u0dba\u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda AadaBelief \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://github.com/juntang-zhuang/Adabelief-Optimizer\">\u0db1\u0dd2\u0dbd \u0dc0\u0dc1\u0dba\u0dd9\u0db1\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a</a> <a href=\"https://arxiv.org/abs/2010.07468\">\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dd2 AadaBelief Optimizer: \u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda \u0dc0\u0dd2\u0dc1\u0dca\u0dc0\u0dcf\u0dc3\u0dba \u0d85\u0db1\u0dd4\u0dc0 \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> . </p>\n<p>\u0db8\u0dd9\u0dba <a href=\"radam.html\">RADAM</a> \u0dc4\u0dd2 \u0daf\u0dd2\u0d9c\u0dd4\u0dc0\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 <a href=\"https://pytorch.org\">PyTorch</a> \u0dc4\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc0\u0dda. </p>\n<p>\u0d86\u0daf\u0db8\u0dca\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dc4 \u0d87\u0da9\u0db6\u0dbd\u0dd2\u0dc6\u0dca \u0d85\u0dad\u0dbb \u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1 \u0dc0\u0dd9\u0db1\u0dc3 \u0db1\u0db8\u0dca, \u0d91\u0dba \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0dd3 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dda \u0d9a\u0dd9\u0dc3\u0dda\u0daf \u0dba\u0db1\u0dca\u0db1\u0dba\u0dd2; \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc0\u0dbb\u0dca\u0d9c \u0dc0\u0dbd on \u0dcf\u0dad\u0dd3\u0dba \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba\u0dd9\u0db1\u0dca \u0db6\u0dd9\u0daf\u0dd3\u0db8 \u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7, \u0d87\u0da9\u0db6\u0dd3\u0dbd\u0dd3\u0dc6\u0dca \u0dc0\u0dd2\u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 on \u0dcf\u0dad\u0dd3\u0dba \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba\u0dd9\u0db1\u0dca \u0db6\u0dd9\u0daf\u0dda. </p>\n<span translate=no>_^_0_^_</span><p>\ud83e\udd14\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2 <span translate=no>_^_1_^_</span>, \u0db1\u0db8\u0dd4\u0dad\u0dca \u0d91\u0dba \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dc5 \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0dba\u0dd0\u0dba\u0dd2 \u0db8\u0da7 \u0dc4\u0dd0\u0d9f\u0dda <span translate=no>_^_2_^_</span>. \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db8\u0dd6\u0dbd\u0dd2\u0d9a \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0dc0\u0dbd\u0dd2\u0db1\u0dca <span translate=no>_^_3_^_</span> \u0db4\u0dc3\u0dd4\u0dc0 \u0dc0\u0db1 \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0db8\u0dd9\u0dba \u0db6\u0ddc\u0dc4\u0ddd \u0daf\u0dda\u0da7 \u0db6\u0dbd\u0db4\u0dcf\u0db1\u0dca\u0db1\u0dda \u0db1\u0dd0\u0dad\u0dd0\u0dba\u0dd2 \u0db8\u0db8 \u0dc3\u0dd2\u0dad\u0db8\u0dd2. </p>\n",
|
||||
"<h2>AdaBelief Optimizer</h2>\n<p>This class extends from RAdam optimizer defined in <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>ADABelief\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0db8\u0dd9\u0db8\u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 RadAM \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dd9\u0db1\u0dca \u0dc0\u0dd2\u0dc4\u0dd2\u0daf\u0dda <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>. </p>\n",
|
||||
"<h3>Calculate <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> or <span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_4_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_5_^_</span> is the current gradient tensor <span translate=no>_^_6_^_</span> for the parameter <span translate=no>_^_7_^_</span></li></ul>\n": "<h3>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 <span translate=no>_^_1_^_</span> \u0dc4\u0ddd <span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_4_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n</ul><li><span translate=no>_^_5_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba <span translate=no>_^_6_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca \u0db5\u0dbd\u0dba \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a tensor \u0dc0\u0dda <span translate=no>_^_7_^_</span></li>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dad\u0dad\u0dca\u0dc0\u0dba\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n</ul><li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_3_^_</span></li>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_16_^_</span> whether to use sgd when the rectification term <span translate=no>_^_17_^_</span> is intractable </li>\n<li><span translate=no>_^_18_^_</span> is whether to use RAdam update </li>\n<li><span translate=no>_^_19_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_20_^_</span>.</li></ul>\n": "<h3>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0dba\u0dd2 </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dba\u0dd2 <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) \u0d9a tuple \u0dc0\u0dda </li>\n<li><span translate=no>_^_6_^_</span> <span translate=no>_^_7_^_</span> \u0dc4\u0ddd \u0db8\u0dad <span translate=no>_^_8_^_</span> \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> <span translate=no>_^_11_^_</span> \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba\u0dda \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0d9a\u0dd2 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d91\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0daf\u0dd9\u0dc0\u0db1 \u0db8\u0ddc\u0dc4\u0ddc\u0dad\u0dda \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db0\u0da2\u0dba\u0d9a\u0dd2 <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> \u0d86\u0daf\u0db8\u0dca \u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf AMSGrad \u0dc4\u0ddd \u0dc0\u0dd0\u0da7\u0dd3\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd9\u0db1 \u0db0\u0da2\u0dba\u0d9a\u0dd2 </li>\n<li><span translate=no>_^_16_^_</span> \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0daf\u0dba <span translate=no>_^_17_^_</span> \u0dc3\u0ddc\u0dba\u0dcf\u0d9c\u0dad \u0db1\u0ddc\u0dc4\u0dd0\u0d9a\u0dd2 \u0dc0\u0dd2\u0da7 sgd \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </li>\n<li><span translate=no>_^_18_^_</span> RaDam \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </li>\n<li><span translate=no>_^_19_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc1\u0db6\u0dca\u0daf \u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dd2. \u0d94\u0db6\u0da7 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dd2\u0da7 \u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dba\u0ddd\u0da2\u0db1\u0dc0\u0dad\u0dca <span translate=no>_^_20_^_</span>\u0dc0\u0dda. </li></ul>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u0daf\u0dd3\u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0d9a\u0dca \u0d9c\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n<li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba <span translate=no>_^_3_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca \u0db5\u0dbd\u0dba \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a tensor \u0dc0\u0dda <span translate=no>_^_4_^_</span> </li>\n</ul><li><span translate=no>_^_5_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_6_^_</span></li>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> otherwise </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc3\u0dc4 <span translate=no>_^_1_^_</span> \u0dc0\u0dd9\u0db1\u0dad\u0dca \u0d86\u0d9a\u0dcf\u0dbb\u0dba\u0d9a\u0dd2\u0db1\u0dca </p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span>. </p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u0db6\u0dbb\u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Difference between gradient and momentum </p>\n": "<p>\u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dc3\u0dc4 \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0d85\u0dad\u0dbb \u0dc0\u0dd9\u0db1\u0dc3 </p>\n",
|
||||
"<p>Exponential moving average of gradient values </p>\n": "<p>\u0db5\u0dbd\u0dba\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc0\u0da7\u0dd2\u0db1\u0dcf\u0d9a\u0db8\u0dca \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba </p>\n",
|
||||
"<p>Exponential moving average of variance </p>\n": "<p>\u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0\u0dba\u0dda\u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dbd\u0db6\u0dcf <span translate=no>_^_0_^_</span> \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span>. </p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> flag is <span translate=no>_^_1_^_</span> for this parameter group, we maintain the maximum of exponential moving average of variance </p>\n": "<p><span translate=no>_^_0_^_</span> \u0db0\u0da2\u0dba \u0db8\u0dd9\u0db8 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dd2\u0dbb\u0dd2\u0dc3\u0d9a\u0dca <span translate=no>_^_1_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0db8\u0dca, \u0d85\u0db4\u0dd2 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0d8b\u0db4\u0dbb\u0dd2\u0db8 \u0db4\u0dc0\u0dad\u0dca\u0dc0\u0dcf </p>\n",
|
||||
"<p>If this parameter group is using <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db8\u0dd9\u0db8\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dda \u0db1\u0db8\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>In-place calculation of <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dba\u0dd9\u0dc4\u0dd2\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 \u0dc0\u0dd0\u0da9\u0dd2 <span translate=no>_^_0_^_</span> \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Maintains max of all exp. moving avg. of sq. grad. values </p>\n": "<p>\u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0d91\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dca\u0dbb\u0dc3\u0dca \u0d8b\u0db4\u0dbb\u0dd2\u0db8 \u0db4\u0dc0\u0dad\u0dca\u0dc0\u0dcf\u0d9c\u0dd9\u0db1 \u0dba\u0dba\u0dd2. \u0d9c\u0db8\u0db1\u0dca Avg. \u0dc0\u0dbb\u0dca\u0d9c. \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda. \u0d85\u0d9c\u0dba\u0db1\u0dca </p>\n",
|
||||
"<p>Perform <em>Adam</em> update, defined in <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>, with <span translate=no>_^_1_^_</span> in place of <span translate=no>_^_2_^_</span>. </p>\n": "<p><em>\u0d86\u0daf\u0db8\u0dca</em> \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>, \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf <span translate=no>_^_1_^_</span> \u0d87\u0dad, \u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7 <span translate=no>_^_2_^_</span>. </p>\n",
|
||||
"<p>Perform <em>Rectified Adam</em> update defined in <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>, with <span translate=no>_^_1_^_</span> in place of <span translate=no>_^_2_^_</span>. </p>\n": "<p><em>\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d86\u0daf\u0db8\u0dca</em> \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>, \u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7 \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf <span translate=no>_^_2_^_</span>\u0d87\u0dad. <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"A simple PyTorch implementation/tutorial of AdaBelief optimizer.": "Adabelief \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dda \u0dc3\u0dbb\u0dbd PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba.",
|
||||
"AdaBelief optimizer": "Atabelief \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1>AdaBelief Optimizer</h1>\n<p>This is based from AdaBelief <a href=\"https://github.com/juntang-zhuang/Adabelief-Optimizer\">official implementation</a> of the paper <a href=\"https://arxiv.org/abs/2010.07468\">AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients</a>.</p>\n<p>This is implemented in <a href=\"https://pytorch.org\">PyTorch</a> as an extension to <a href=\"radam.html\">RAdam</a>.</p>\n<p>The main difference between Adam optimizer and AdaBelief is that, how it calculates the adaptive learning rate; instead of dividing by the exponential moving average of square of the gradients, AdaBelief divides by the exponential mean of variance.</p>\n<span translate=no>_^_0_^_</span><p>\ud83e\udd14 The paper calculates variance as <span translate=no>_^_1_^_</span>, but I feel it should use the bias corrected momentum <span translate=no>_^_2_^_</span>. I guess this doesn't affect things much because bias correction is <span translate=no>_^_3_^_</span> after the initial training steps.</p>\n": "<h1>adaBelief \u4f18\u5316\u5668</h1>\n<p>\u8fd9\u662f\u57fa\u4e8e AdaBeLief Optimizer \u8bba\u6587<a href=\"https://github.com/juntang-zhuang/Adabelief-Optimizer\">\u300a<a href=\"https://arxiv.org/abs/2010.07468\">AdaBeLief Optimizer\uff1a\u901a\u8fc7\u5bf9\u89c2\u5bdf\u5230\u7684\u68af\u5ea6\u7684\u4fe1\u5ff5\u8c03\u6574\u6b65\u957f\u300b</a>\u7684\u5b98\u65b9\u5b9e\u73b0</a>\u3002</p>\n<p>\u8fd9\u662f\u5728 <a href=\"https://pytorch.org\">PyTorch</a> \u4e2d\u4f5c\u4e3a\u5bf9 <a href=\"radam.html\">RadAM</a> \u7684\u6269\u5c55\u5b9e\u73b0\u7684\u3002</p>\n<p>Adam optimizer \u548c AdaBeLief \u4e4b\u95f4\u7684\u4e3b\u8981\u533a\u522b\u5728\u4e8e\uff0c\u5b83\u5982\u4f55\u8ba1\u7b97\u81ea\u9002\u5e94\u5b66\u4e60\u7387\uff1bAdaBeLief \u4e0d\u662f\u9664\u4ee5\u68af\u5ea6\u5e73\u65b9\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u503c\uff0c\u800c\u662f\u9664\u4ee5\u65b9\u5dee\u7684\u6307\u6570\u5747\u503c\u3002</p>\n<span translate=no>_^_0_^_</span><p>\ud83e\udd14 \u672c\u6587\u5c06\u65b9\u5dee\u8ba1\u7b97\u4e3a<span translate=no>_^_1_^_</span>\uff0c\u4f46\u6211\u8ba4\u4e3a\u5b83\u5e94\u8be5\u4f7f\u7528\u504f\u5dee\u6821\u6b63\u7684\u52a8\u91cf<span translate=no>_^_2_^_</span>\u3002\u6211\u60f3\u8fd9\u5bf9\u4e8b\u60c5\u7684\u5f71\u54cd\u4e0d\u5927\uff0c\u56e0\u4e3a\u504f\u5dee\u6821\u6b63\u662f\u5728\u6700\u521d\u7684\u8bad\u7ec3\u6b65\u9aa4<span translate=no>_^_3_^_</span>\u4e4b\u540e\u8fdb\u884c\u7684\u3002</p>\n",
|
||||
"<h2>AdaBelief Optimizer</h2>\n<p>This class extends from RAdam optimizer defined in <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>adaBelief \u4f18\u5316\u5668</h2>\n<p>\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 RadAM \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684<a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>\u3002</p>\n",
|
||||
"<h3>Calculate <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> or <span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_4_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_5_^_</span> is the current gradient tensor <span translate=no>_^_6_^_</span> for the parameter <span translate=no>_^_7_^_</span></li></ul>\n": "<h3>\u8ba1\u7b97<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span>\u6216<span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_4_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_5_^_</span>\u662f\u53c2\u6570\u7684\u5f53\u524d\u68af<span translate=no>_^_6_^_</span>\u5ea6\u5f20\u91cf<span translate=no>_^_7_^_</span></li></ul>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>\u521d\u59cb\u5316\u53c2\u6570\u72b6\u6001</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_16_^_</span> whether to use sgd when the rectification term <span translate=no>_^_17_^_</span> is intractable </li>\n<li><span translate=no>_^_18_^_</span> is whether to use RAdam update </li>\n<li><span translate=no>_^_19_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_20_^_</span>.</li></ul>\n": "<h3>\u521d\u59cb\u5316\u4f18\u5316\u5668</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\u5217\u8868</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5b66\u4e60\u7387<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f (<span translate=no>_^_4_^_</span>,<span translate=no>_^_5_^_</span>) \u7684\u5143\u7ec4</li>\n<li><span translate=no>_^_6_^_</span>\u662f<span translate=no>_^_7_^_</span>\u6216<span translate=no>_^_8_^_</span>\u57fa\u4e8e<span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span>\u662f\u5728\u4e2d<span translate=no>_^_11_^_</span>\u5b9a\u4e49\u7684\u7c7b\u7684\u5b9e\u4f8b <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li><span translate=no>_^_13_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u662f\u5426\u5728\u6dfb\u52a0\u540e\u901a\u8fc7\u8fd9\u6837\u505a\u6765\u4f18\u5316\u7b2c\u4e8c\u4e2a\u65f6\u523b\u7684\u504f\u5dee\u6821\u6b63<span translate=no>_^_14_^_</span></li>\n<li><span translate=no>_^_15_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u6307\u793a\u662f\u4f7f\u7528 AmsGrad \u8fd8\u662f\u56de\u9000\u5230\u666e\u901a\u7684 Adam</li>\n<li><span translate=no>_^_16_^_</span>\u7ea0\u6b63\u6761\u6b3e<span translate=no>_^_17_^_</span>\u96be\u4ee5\u5904\u7406\u65f6\u662f\u5426\u4f7f\u7528 sgd</li>\n<li><span translate=no>_^_18_^_</span>\u662f\u5426\u4f7f\u7528 raDAM \u66f4\u65b0</li>\n<li><span translate=no>_^_19_^_</span>\u662f\u7ec4\u503c\u7684\u9ed8\u8ba4\u5b57\u5178\u3002\u5f53\u4f60\u60f3\u6269\u5c55\u7c7b\u65f6\uff0c\u8fd9\u5f88\u6709\u7528<span translate=no>_^_20_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u5bf9\u7ed9\u5b9a\u53c2\u6570\u5f20\u91cf\u6267\u884c\u66f4\u65b0\u6b65\u9aa4</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u7684\u5f53\u524d\u68af<span translate=no>_^_3_^_</span>\u5ea6\u5f20\u91cf<span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> otherwise </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5426\u5219</p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u8ba1\u7b97<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u8ba1\u7b97\u4f53\u91cd\u8870\u51cf</p>\n",
|
||||
"<p>Difference between gradient and momentum </p>\n": "<p>\u68af\u5ea6\u548c\u52a8\u91cf\u4e4b\u95f4\u7684\u533a\u522b</p>\n",
|
||||
"<p>Exponential moving average of gradient values </p>\n": "<p>\u68af\u5ea6\u503c\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf</p>\n",
|
||||
"<p>Exponential moving average of variance </p>\n": "<p>\u65b9\u5dee\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf</p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u83b7\u53d6<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u5f97\u5230<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> flag is <span translate=no>_^_1_^_</span> for this parameter group, we maintain the maximum of exponential moving average of variance </p>\n": "<p>\u5982\u679c f<span translate=no>_^_0_^_</span> lag<span translate=no>_^_1_^_</span> \u7528\u4e8e\u6b64\u53c2\u6570\u7ec4\uff0c\u5219\u6211\u4eec\u7ef4\u6301\u65b9\u5dee\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u7684\u6700\u5927\u503c</p>\n",
|
||||
"<p>If this parameter group is using <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u6b64\u53c2\u6570\u7ec4\u6b63\u5728\u4f7f\u7528<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>In-place calculation of <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5c31\u5730\u8ba1\u7b97<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p><span translate=no>_^_0_^_</span>\u589e\u52a0\u4f18\u5316\u5668\u6b65\u6570</p>\n",
|
||||
"<p>Maintains max of all exp. moving avg. of sq. grad. values </p>\n": "<p>\u4fdd\u6301\u6240\u6709 exp. \u79fb\u52a8\u5e73\u5747 sq. grad. \u503c\u7684\u6700\u5927\u503c</p>\n",
|
||||
"<p>Perform <em>Adam</em> update, defined in <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>, with <span translate=no>_^_1_^_</span> in place of <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u6267\u884c <em>Adam</em> \u66f4\u65b0\uff0c\u5728\u4e2d\u5b9a\u4e49 <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>\uff0c\u7528<span translate=no>_^_1_^_</span>\u4ee3\u66ff<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"<p>Perform <em>Rectified Adam</em> update defined in <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a>, with <span translate=no>_^_1_^_</span> in place of <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u6267\u884c\u4e2d\u5b9a\u4e49\u7684\u5df2<em>\u6821\u6b63\u7684 Adam</em> \u66f4\u65b0 <a href=\"radam.html\"><span translate=no>_^_0_^_</span></a><span translate=no>_^_1_^_</span>\uff0c\u7528\u4ee3\u66ff<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of AdaBelief optimizer.": "AdaBeLief \u4f18\u5316\u5668\u7684\u7b80\u5355\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002",
|
||||
"AdaBelief optimizer": "adaBeLief \u4f18\u5316\u5668"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1>Adam Optimizer</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of popular optimizer <em>Adam</em> from paper <a href=\"https://arxiv.org/abs/1412.6980\">Adam: A Method for Stochastic Optimization</a>.</p>\n<p><em>Adam</em> update is,</p>\n<span translate=no>_^_0_^_</span><p>where <span translate=no>_^_1_^_</span>, <span translate=no>_^_2_^_</span>, <span translate=no>_^_3_^_</span> and <span translate=no>_^_4_^_</span> are scalar hyper parameters. <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> are first and second order moments. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span> are biased corrected moments. <span translate=no>_^_9_^_</span> is used as a fix for division by zero error, but also acts as a form of a hyper-parameter that acts against variance in gradients.</p>\n<p>Effective step taken assuming <span translate=no>_^_10_^_</span> is, <span translate=no>_^_11_^_</span> This is bounded by, <span translate=no>_^_12_^_</span> when <span translate=no>_^_13_^_</span> and <span translate=no>_^_14_^_</span> otherwise. And in most common scenarios, <span translate=no>_^_15_^_</span></p>\n": "<h1>\u30a2\u30c0\u30e0\u30fb\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<em>\u30a2\u30c0\u30e0<a href=\"https://arxiv.org/abs/1412.6980\">\uff1a\u78ba\u7387\u7684\u6700\u9069\u5316\u306e\u65b9\u6cd5\u300d<a href=\"https://pytorch.org\">\u306b\u63b2\u8f09\u3055\u308c\u305f\u4eba\u6c17\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fcAdam\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a></em>\u3002</p>\n<p><em>\u30a2\u30c0\u30e0\u306e\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u306f</em>\u3001</p>\n<span translate=no>_^_0_^_</span><p>\u3053\u3053\u3067<span translate=no>_^_1_^_</span>\u3001<span translate=no>_^_2_^_</span>\u3001<span translate=no>_^_3_^_</span><span translate=no>_^_4_^_</span>\u304a\u3088\u3073\u306f\u30b9\u30ab\u30e9\u30fc\u306e\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3067\u3059\u3002<span translate=no>_^_5_^_</span>\u30d5\u30a1\u30fc\u30b9\u30c8\u30aa\u30fc\u30c0\u30fc\u3001\u30bb\u30ab\u30f3\u30c9\u30aa\u30fc\u30c0\u30fc\u306e\u77ac\u9593\u3067\u3059 <span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span>\u504f\u308a\u4fee\u6b63\u3055\u308c\u305f\u30e2\u30fc\u30e1\u30f3\u30c8\u3067\u3059\u3002<span translate=no>_^_9_^_</span>\u30bc\u30ed\u30a8\u30e9\u30fc\u306b\u3088\u308b\u9664\u7b97\u306e\u4fee\u6b63\u3068\u3057\u3066\u4f7f\u308f\u308c\u307e\u3059\u304c\u3001\u52fe\u914d\u306e\u3070\u3089\u3064\u304d\u306b\u5bfe\u3057\u3066\u4f5c\u7528\u3059\u308b\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u5f62\u5f0f\u3068\u3057\u3066\u3082\u6a5f\u80fd\u3057\u307e\u3059</p>\u3002\n<p><span translate=no>_^_10_^_</span>\u6709\u52b9\u306a\u624b\u9806\u306f\u3001\u300c<span translate=no>_^_11_^_</span>This \u304c\u5236\u9650\u3055\u308c\u308b\u300d\u3001\u300c\u3044\u3064\u300d\u3001\u300c<span translate=no>_^_12_^_</span>\u305d\u308c\u4ee5\u5916\u306e\u5834\u5408<span translate=no>_^_13_^_</span>\u300d\u3092\u524d\u63d0\u3068\u3057\u3066\u3044\u307e\u3059\u3002<span translate=no>_^_14_^_</span>\u305d\u3057\u3066\u3001\u6700\u3082\u4e00\u822c\u7684\u306a\u30b7\u30ca\u30ea\u30aa\u3067\u306f\u3001<span translate=no>_^_15_^_</span></p>\n",
|
||||
"<h2>Adam Optimizer</h2>\n<p>We extend the class <span translate=no>_^_0_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_1_^_</span></a> to implement the Adam optimizer.</p>\n": "<h2>\u30a2\u30c0\u30e0\u30fb\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h2>\n<p><span translate=no>_^_0_^_</span><a href=\"index.html\"><span translate=no>_^_1_^_</span></a>\u3067\u5b9a\u7fa9\u3057\u305f\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3057\u3066 Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u5b9f\u88c5\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Calculate <span translate=no>_^_0_^_</span> and and <span translate=no>_^_1_^_</span></h3>\n<ul><li><span translate=no>_^_2_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_3_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_4_^_</span> is the current gradient tensor <span translate=no>_^_5_^_</span> for the parameter <span translate=no>_^_6_^_</span></li></ul>\n": "<h3><span translate=no>_^_0_^_</span>\u8a08\u7b97\u3068 <span translate=no>_^_1_^_</span></h3>\n<ul><li><span translate=no>_^_2_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_3_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u73fe\u5728\u306e\u52fe\u914d\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<h3>Do the <em>Adam</em> parameter update</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are the uncorrected first and second moments <span translate=no>_^_6_^_</span> and <span translate=no>_^_7_^_</span>.</li></ul>\n<p>This computes the following</p>\n<span translate=no>_^_8_^_</span><p>Since <span translate=no>_^_9_^_</span>, <span translate=no>_^_10_^_</span>, <span translate=no>_^_11_^_</span> and <span translate=no>_^_12_^_</span> are scalars and others are tensors we modify this calculation to optimize the computation.</p>\n<span translate=no>_^_13_^_</span><p>where <span translate=no>_^_14_^_</span> is what we should specify as the hyper-parameter.</p>\n": "<h3><em>Adam</em> \u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u66f4\u65b0\u3057\u3066\u304f\u3060\u3055\u3044</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span><span translate=no>_^_6_^_</span>\u305d\u3057\u3066\u672a\u4fee\u6b63\u306e\u7b2c\u4e00\u77ac\u9593\u3068\u7b2c\u4e8c\u77ac\u9593\u3068 <span translate=no>_^_7_^_</span></li></ul>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001\u4ee5\u4e0b\u304c\u8a08\u7b97\u3055\u308c\u307e\u3059</p>\n<span translate=no>_^_8_^_</span><p><span translate=no>_^_9_^_</span><span translate=no>_^_10_^_</span>\u3001<span translate=no>_^_11_^_</span><span translate=no>_^_12_^_</span>\u306f\u30b9\u30ab\u30e9\u30fc\u3067\u3001\u305d\u306e\u4ed6\u306f\u30c6\u30f3\u30bd\u30eb\u306a\u306e\u3067\u3001\u3053\u306e\u8a08\u7b97\u3092\u5909\u66f4\u3057\u3066\u8a08\u7b97\u3092\u6700\u9069\u5316\u3057\u307e\u3059\u3002</p>\n<span translate=no>_^_13_^_</span><p><span translate=no>_^_14_^_</span>\u3053\u3053\u3067\u3001\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u3068\u3057\u3066\u6307\u5b9a\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p>This returns the modified learning rate based on the state. For <em>Adam</em> this is just the specified learning rate for the parameter group, <span translate=no>_^_0_^_</span>.</p>\n": "<h3>\u5b66\u7fd2\u7387\u3092\u53d6\u5f97</h3>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001\u72b6\u614b\u306b\u57fa\u3065\u3044\u3066\u4fee\u6b63\u3055\u308c\u305f\u5b66\u7fd2\u7387\u304c\u8fd4\u3055\u308c\u307e\u3059\u3002<em>Adam</em> \u306e\u5834\u5408\u3001\u3053\u308c\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306b\u6307\u5b9a\u3055\u308c\u3066\u3044\u308b\u5b66\u7fd2\u7387\u306b\u3059\u304e\u307e\u305b\u3093<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>\u30d1\u30e9\u30e1\u30fc\u30bf\u72b6\u614b\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_16_^_</span>.</li></ul>\n": "<h3>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u30ea\u30b9\u30c8\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u5b66\u7fd2\u7387 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>(,) <span translate=no>_^_4_^_</span> \u306e\u30bf\u30d7\u30eb\u3067\u3059 <span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span>\u307e\u305f\u306f\u305d\u308c\u306b\u57fa\u3065\u3044\u3066\u3044\u308b <span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span><span translate=no>_^_11_^_</span>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3067\u3059 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li><span translate=no>_^_13_^_</span>\u30bb\u30ab\u30f3\u30c9\u30e2\u30fc\u30e1\u30f3\u30c8\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3092\u52a0\u7b97\u3057\u3066\u304b\u3089\u884c\u3046\u3053\u3068\u3067\u6700\u9069\u5316\u3059\u308b\u304b\u5426\u304b\u306e\u30d5\u30e9\u30b0\u3067\u3059 <span translate=no>_^_14_^_</span></li>\n<li><span translate=no>_^_15_^_</span>\u30b0\u30eb\u30fc\u30d7\u5024\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u8f9e\u66f8\u3067\u3059\u3002\u3053\u308c\u306f\u3001\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3059\u308b\u5834\u5408\u306b\u4fbf\u5229\u3067\u3059<span translate=no>_^_16_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u4e0e\u3048\u3089\u308c\u305f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb\u306e\u66f4\u65b0\u30b9\u30c6\u30c3\u30d7\u3092\u5b9f\u884c\u3059\u308b</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u73fe\u5728\u306e\u52fe\u914d\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Bias correction term for <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u7528\u8a9e <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u4f53\u91cd\u6e1b\u5c11\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Computation without optimization </p>\n": "<p>\u6700\u9069\u5316\u306a\u3057\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Exponential moving average of gradients, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u52fe\u914d\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3001<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Exponential moving average of squared gradient values, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4e8c\u4e57\u52fe\u914d\u5024\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3001<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u53d6\u5f97\u3057\u3066 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get learning rate </p>\n": "<p>\u5b66\u7fd2\u7387\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>In-place calculation of <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n": "<p>\u306e\u30a4\u30f3\u30d7\u30ec\u30fc\u30b9\u8a08\u7b97 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p><span translate=no>_^_0_^_</span>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u30b9\u30c6\u30c3\u30d7\u6570\u3092\u5897\u3084\u3059</p>\n",
|
||||
"<p>Perform <em>Adam</em> update </p>\n": "<p><em>Adam</em> \u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>This is the number of optimizer steps taken on the parameter, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3053\u308c\u306f\u3001\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u306b\u5bfe\u3057\u3066\u5b9f\u884c\u3055\u308c\u305f\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u30b9\u30c6\u30c3\u30d7\u306e\u6570\u3067\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Whether to optimize the computation </p>\n": "<p>\u8a08\u7b97\u3092\u6700\u9069\u5316\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer": "Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u7c21\u5358\u306a PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb",
|
||||
"Adam Optimizer": "\u30a2\u30c0\u30e0\u30fb\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1>Adam Optimizer</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of popular optimizer <em>Adam</em> from paper <a href=\"https://arxiv.org/abs/1412.6980\">Adam: A Method for Stochastic Optimization</a>.</p>\n<p><em>Adam</em> update is,</p>\n<span translate=no>_^_0_^_</span><p>where <span translate=no>_^_1_^_</span>, <span translate=no>_^_2_^_</span>, <span translate=no>_^_3_^_</span> and <span translate=no>_^_4_^_</span> are scalar hyper parameters. <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> are first and second order moments. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span> are biased corrected moments. <span translate=no>_^_9_^_</span> is used as a fix for division by zero error, but also acts as a form of a hyper-parameter that acts against variance in gradients.</p>\n<p>Effective step taken assuming <span translate=no>_^_10_^_</span> is, <span translate=no>_^_11_^_</span> This is bounded by, <span translate=no>_^_12_^_</span> when <span translate=no>_^_13_^_</span> and <span translate=no>_^_14_^_</span> otherwise. And in most common scenarios, <span translate=no>_^_15_^_</span></p>\n": "<h1>\u0d86\u0daf\u0db8\u0dca\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 \u0da2\u0db1\u0db4\u0dca\u0dbb\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <em>\u0d87\u0da9\u0db8\u0dca</em> \u0dc0\u0dd9\u0dad\u0dd2\u0db1\u0dca <a href=\"https://arxiv.org/abs/1412.6980\">\u0d87\u0da9\u0db8\u0dca: \u0dc3\u0dca\u0da7\u0ddc\u0da0\u0dcf\u0dc3\u0dca\u0da7\u0dd2\u0d9a\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dca\u0dbb\u0db8\u0dba\u0d9a\u0dca</a> . </p>\n<p><em>\u0d86\u0daf\u0db8\u0dca</em> \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8,</p>\n<span translate=no>_^_0_^_</span><p>\u0d9a\u0ddc\u0dc4\u0dd9\u0daf <span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>, <span translate=no>_^_3_^_</span> \u0dc3\u0dc4 <span translate=no>_^_4_^_</span> \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab \u0d85\u0db0\u0dd2 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc0\u0dda. <span translate=no>_^_5_^_</span> \u0dc3\u0dc4 \u0db4\u0dc5\u0db8\u0dd4 <span translate=no>_^_6_^_</span> \u0dc4\u0dcf \u0daf\u0dd9\u0dc0\u0db1 \u0d87\u0dab\u0dc0\u0dd4\u0db8\u0dca \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf \u0dc0\u0dda. <span translate=no>_^_7_^_</span> <span translate=no>_^_8_^_</span> \u0dc3\u0dc4 \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0db1\u0dca \u0dc0\u0dda. <span translate=no>_^_9_^_</span> \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0daf\u0ddd\u0dc2\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0db6\u0dd9\u0daf\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0dc3\u0db3\u0dd4\u0db8\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2, \u0db1\u0db8\u0dd4\u0dad\u0dca \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc0\u0dd2\u0da0\u0dbd\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba\u0da7 \u0d91\u0dbb\u0dd9\u0dc4\u0dd2\u0dc0 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf \u0d9a\u0dbb\u0db1 \u0d85\u0db0\u0dd2-\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba\u0d9a \u0d86\u0d9a\u0dcf\u0dbb\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3\u0daf \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf \u0d9a\u0dbb\u0dba\u0dd2. </p>\n<p>\u0d8b\u0db4\u0d9a\u0dbd\u0dca\u0db4\u0db1\u0dba\u0d9a\u0dbb\u0db8\u0dd2\u0db1\u0dca \u0d9c\u0db1\u0dca\u0db1\u0dcf \u0dbd\u0daf effective \u0dbd\u0daf\u0dcf\u0dba\u0dd3 \u0db4\u0dd2\u0dba\u0dc0\u0dbb <span translate=no>_^_10_^_</span> \u0dc0\u0db1\u0dca\u0db1\u0dda, <span translate=no>_^_11_^_</span> \u0db8\u0dd9\u0dba \u0db8\u0dcf\u0dba\u0dd2\u0db8\u0dca \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0db6\u0db1\u0dca\u0db1\u0dda <span translate=no>_^_12_^_</span> <span translate=no>_^_13_^_</span> \u0d9a\u0dc0\u0daf\u0dcf\u0daf \u0dc3\u0dc4 <span translate=no>_^_14_^_</span> \u0dc0\u0dd9\u0db1\u0dad\u0dca \u0d86\u0d9a\u0dcf\u0dbb\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dba. \u0dc3\u0dc4 \u0dc0\u0da9\u0dcf\u0dad\u0dca \u0db4\u0ddc\u0daf\u0dd4 \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0db1\u0dca\u0dc4\u0dd3\u0daf\u0dd3, <span translate=no>_^_15_^_</span></p>\n",
|
||||
"<h2>Adam Optimizer</h2>\n<p>We extend the class <span translate=no>_^_0_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_1_^_</span></a> to implement the Adam optimizer.</p>\n": "<h2>\u0d86\u0daf\u0db8\u0dca\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0d86\u0daf\u0db8\u0dca\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"index.html\"><span translate=no>_^_1_^_</span></a> \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 <span translate=no>_^_0_^_</span> \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
|
||||
"<h3>Calculate <span translate=no>_^_0_^_</span> and and <span translate=no>_^_1_^_</span></h3>\n<ul><li><span translate=no>_^_2_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_3_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_4_^_</span> is the current gradient tensor <span translate=no>_^_5_^_</span> for the parameter <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 <span translate=no>_^_1_^_</span></h3>\n<ul><li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_3_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n</ul><li><span translate=no>_^_4_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba <span translate=no>_^_5_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca \u0db5\u0dbd\u0dba \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a tensor \u0dc0\u0dda <span translate=no>_^_6_^_</span></li>\n",
|
||||
"<h3>Do the <em>Adam</em> parameter update</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are the uncorrected first and second moments <span translate=no>_^_6_^_</span> and <span translate=no>_^_7_^_</span>.</li></ul>\n<p>This computes the following</p>\n<span translate=no>_^_8_^_</span><p>Since <span translate=no>_^_9_^_</span>, <span translate=no>_^_10_^_</span>, <span translate=no>_^_11_^_</span> and <span translate=no>_^_12_^_</span> are scalars and others are tensors we modify this calculation to optimize the computation.</p>\n<span translate=no>_^_13_^_</span><p>where <span translate=no>_^_14_^_</span> is what we should specify as the hyper-parameter.</p>\n": "<h3><em>\u0d86\u0daf\u0db8\u0dca</em> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n<li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> <span translate=no>_^_5_^_</span> \u0dc3\u0dc4 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0db1\u0ddc\u0d9a\u0dc5 \u0db4\u0dc5\u0db8\u0dd4 \u0dc4\u0dcf \u0daf\u0dd9\u0dc0\u0db1 \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf <span translate=no>_^_6_^_</span> \u0dc3\u0dc4 <span translate=no>_^_7_^_</span>. </li></ul>\n<p>\u0db8\u0dd9\u0dba\u0db4\u0dc4\u0dad \u0dc3\u0db3\u0dc4\u0db1\u0dca \u0daf\u0dda \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2</p>\n<span translate=no>_^_8_^_</span><p>\u0dc3\u0dd2\u0da7 <span translate=no>_^_9_^_</span><span translate=no>_^_10_^_</span>, <span translate=no>_^_11_^_</span> \u0dc3\u0dc4 \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0db1\u0dca <span translate=no>_^_12_^_</span> \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0d85\u0db1\u0dd9\u0d9a\u0dca \u0d92\u0dc0\u0dcf \u0d85\u0db4\u0dd2 \u0db8\u0dd9\u0db8 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dbb\u0db1 \u0d86\u0dad\u0dad\u0dd3\u0db1\u0dca \u0dc0\u0dda \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n<span translate=no>_^_13_^_</span><p>\u0d85\u0db0\u0dd2-\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba\u0dbd\u0dd9\u0dc3 \u0d85\u0db4 \u0dc3\u0db3\u0dc4\u0db1\u0dca \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dca\u0dad\u0dda <span translate=no>_^_14_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p>This returns the modified learning rate based on the state. For <em>Adam</em> this is just the specified learning rate for the parameter group, <span translate=no>_^_0_^_</span>.</p>\n": "<h3>\u0d89\u0d9c\u0dd9\u0db1\u0dd3\u0db8-\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1</h3>\n<p>\u0db8\u0dd9\u0dba\u0dbb\u0dcf\u0da2\u0dca\u0dba\u0dba \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca\u0dc0 \u0db1\u0dc0\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0dba\u0dd2. <em>\u0d86\u0daf\u0db8\u0dca</em> \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dd9\u0dba \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dc1\u0dca\u0da0\u0dd2\u0dad \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0db4\u0db8\u0dab\u0dd2, <span translate=no>_^_0_^_</span>. </p>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dad\u0dad\u0dca\u0dc0\u0dba\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n</ul><li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_3_^_</span></li>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_16_^_</span>.</li></ul>\n": "<h3>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0dba\u0dd2 </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dba\u0dd2 <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) \u0d9a tuple \u0dc0\u0dda </li>\n<li><span translate=no>_^_6_^_</span> <span translate=no>_^_7_^_</span> \u0dc4\u0ddd \u0db8\u0dad <span translate=no>_^_8_^_</span> \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> <span translate=no>_^_11_^_</span> \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba\u0dda \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0d9a\u0dd2 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d91\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0daf\u0dd9\u0dc0\u0db1 \u0db8\u0ddc\u0dc4\u0ddc\u0dad\u0dda \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db0\u0da2\u0dba\u0d9a\u0dd2 <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc1\u0db6\u0dca\u0daf \u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dd2. \u0d94\u0db6\u0da7 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dd2\u0da7 \u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dba\u0ddd\u0da2\u0db1\u0dc0\u0dad\u0dca <span translate=no>_^_16_^_</span>\u0dc0\u0dda. </li></ul>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u0daf\u0dd3\u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0d9a\u0dca \u0d9c\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n<li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba <span translate=no>_^_3_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca \u0db5\u0dbd\u0dba \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a tensor \u0dc0\u0dda <span translate=no>_^_4_^_</span> </li>\n</ul><li><span translate=no>_^_5_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_6_^_</span></li>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Bias correction term for <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0db3\u0dc4\u0dcf\u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0daf\u0dba <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u0db6\u0dbb\u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Computation without optimization </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dd2\u0db1\u0dca\u0dad\u0ddc\u0dbb\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Exponential moving average of gradients, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9a \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Exponential moving average of squared gradient values, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc0\u0dbb\u0dca\u0d9c\u0db5\u0dbd\u0dba \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc0\u0da7\u0dd2\u0db1\u0dcf\u0d9a\u0db8\u0dca \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dbd\u0db6\u0dcf <span translate=no>_^_0_^_</span> \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Get learning rate </p>\n": "<p>\u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>In-place calculation of <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dba\u0dd9\u0dc4\u0dd2\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 \u0dc0\u0dd0\u0da9\u0dd2 <span translate=no>_^_0_^_</span> \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Perform <em>Adam</em> update </p>\n": "<p><em>\u0d86\u0daf\u0db8\u0dca</em> \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dd2\u0daf\u0dd4 </p>\n",
|
||||
"<p>This is the number of optimizer steps taken on the parameter, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba\u0db8\u0dad \u0d9c\u0dd9\u0db1 \u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 \u0db8\u0dd9\u0dba\u0dba\u0dd2, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Whether to optimize the computation </p>\n": "<p>\u0db8\u0dd9\u0db8\u0d9c\u0dab\u0db1\u0dba \u0d8b\u0db4\u0dbb\u0dd2\u0db8 \u0db5\u0dbd \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer": "\u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dcf\u0dbb\u0d9a\u0dba\u0dda \u0dc3\u0dbb\u0dbd \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba",
|
||||
"Adam Optimizer": "\u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1>Adam Optimizer</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of popular optimizer <em>Adam</em> from paper <a href=\"https://arxiv.org/abs/1412.6980\">Adam: A Method for Stochastic Optimization</a>.</p>\n<p><em>Adam</em> update is,</p>\n<span translate=no>_^_0_^_</span><p>where <span translate=no>_^_1_^_</span>, <span translate=no>_^_2_^_</span>, <span translate=no>_^_3_^_</span> and <span translate=no>_^_4_^_</span> are scalar hyper parameters. <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> are first and second order moments. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span> are biased corrected moments. <span translate=no>_^_9_^_</span> is used as a fix for division by zero error, but also acts as a form of a hyper-parameter that acts against variance in gradients.</p>\n<p>Effective step taken assuming <span translate=no>_^_10_^_</span> is, <span translate=no>_^_11_^_</span> This is bounded by, <span translate=no>_^_12_^_</span> when <span translate=no>_^_13_^_</span> and <span translate=no>_^_14_^_</span> otherwise. And in most common scenarios, <span translate=no>_^_15_^_</span></p>\n": "<h1>\u4e9a\u5f53\u4f18\u5316\u5668</h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a<em>\u4e9a</em>\u5f53<a href=\"https://arxiv.org/abs/1412.6980\">\uff1a\u968f\u673a\u4f18\u5316\u65b9\u6cd5\u300b\u4e2d\u6d41\u884c\u7684\u4f18\u5316\u5668 Adam \u7684 <a href=\"https://pytorch.org\">Py</a> Torch</a> \u5b9e\u73b0\u3002</p>\n<p><em>\u4e9a\u5f53</em>\u7684\u66f4\u65b0\u662f\uff0c</p>\n<span translate=no>_^_0_^_</span><p>\u5176\u4e2d<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u3001<span translate=no>_^_3_^_</span>\u548c<span translate=no>_^_4_^_</span>\u662f\u6807\u91cf\u8d85\u7ea7\u53c2\u6570\u3002<span translate=no>_^_5_^_</span>\u548c<span translate=no>_^_6_^_</span>\u662f\u4e00\u9636\u548c\u4e8c\u9636\u65f6\u523b\u3002<span translate=no>_^_7_^_</span>\u5e76\u4e14<span translate=no>_^_8_^_</span>\u662f\u6709\u504f\u5dee\u7684\u6821\u6b63\u65f6\u523b\u3002<span translate=no>_^_9_^_</span>\u7528\u4f5c\u9664\u4ee5\u96f6\u8bef\u5dee\u7684\u4fee\u590d\uff0c\u4f46\u4e5f\u7528\u4f5c\u5bf9\u68af\u5ea6\u65b9\u5dee\u8d77\u4f5c\u7528\u7684\u8d85\u53c2\u6570\u7684\u4e00\u79cd\u5f62\u5f0f\u3002</p>\n<p>\u5047\u8bbe\u91c7\u53d6\u7684\u6709\u6548\u6b65\u9aa4<span translate=no>_^_10_^_</span>\u662f\uff0c<span translate=no>_^_11_^_</span>\u8fd9\u53d7\u9650\u4e8e\u3001<span translate=no>_^_12_^_</span>\u4f55\u65f6<span translate=no>_^_13_^_</span>\u4ee5\u53ca<span translate=no>_^_14_^_</span>\u5176\u4ed6\u65b9\u9762\u3002\u5728\u5927\u591a\u6570\u5e38\u89c1\u60c5\u51b5\u4e0b\uff0c<span translate=no>_^_15_^_</span></p>\n",
|
||||
"<h2>Adam Optimizer</h2>\n<p>We extend the class <span translate=no>_^_0_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_1_^_</span></a> to implement the Adam optimizer.</p>\n": "<h2>\u4e9a\u5f53\u4f18\u5316\u5668</h2>\n<p>\u6211\u4eec\u6269\u5c55\u4e86\u4e2d<span translate=no>_^_0_^_</span>\u5b9a\u4e49\u7684\u7c7b<a href=\"index.html\"><span translate=no>_^_1_^_</span></a>\u6765\u5b9e\u73b0 Adam \u4f18\u5316\u5668\u3002</p>\n",
|
||||
"<h3>Calculate <span translate=no>_^_0_^_</span> and and <span translate=no>_^_1_^_</span></h3>\n<ul><li><span translate=no>_^_2_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_3_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_4_^_</span> is the current gradient tensor <span translate=no>_^_5_^_</span> for the parameter <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u8ba1\u7b97<span translate=no>_^_0_^_</span>\u548c\u548c<span translate=no>_^_1_^_</span></h3>\n<ul><li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_3_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_4_^_</span>\u662f\u53c2\u6570\u7684\u5f53\u524d\u68af<span translate=no>_^_5_^_</span>\u5ea6\u5f20\u91cf<span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<h3>Do the <em>Adam</em> parameter update</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are the uncorrected first and second moments <span translate=no>_^_6_^_</span> and <span translate=no>_^_7_^_</span>.</li></ul>\n<p>This computes the following</p>\n<span translate=no>_^_8_^_</span><p>Since <span translate=no>_^_9_^_</span>, <span translate=no>_^_10_^_</span>, <span translate=no>_^_11_^_</span> and <span translate=no>_^_12_^_</span> are scalars and others are tensors we modify this calculation to optimize the computation.</p>\n<span translate=no>_^_13_^_</span><p>where <span translate=no>_^_14_^_</span> is what we should specify as the hyper-parameter.</p>\n": "<h3><em>Adam</em> \u53c2\u6570\u662f\u5426\u66f4\u65b0</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u5e76\u4e14<span translate=no>_^_5_^_</span>\u662f\u672a\u6821\u6b63\u7684\u7b2c\u4e00\u548c\u7b2c\u4e8c\u65f6\u523b<span translate=no>_^_6_^_</span>\uff0c\u4ee5\u53ca<span translate=no>_^_7_^_</span>.</li></ul>\n<p>\u8fd9\u8ba1\u7b97\u51fa\u4ee5\u4e0b\u5185\u5bb9</p>\n<span translate=no>_^_8_^_</span>\u7531<p>\u4e8e<span translate=no>_^_9_^_</span><span translate=no>_^_10_^_</span>\u3001<span translate=no>_^_11_^_</span>\u548c<span translate=no>_^_12_^_</span>\u662f\u6807\u91cf\uff0c\u5176\u4ed6\u662f\u5f20\u91cf\uff0c\u56e0\u6b64\u6211\u4eec\u5c06\u6b64\u8ba1\u7b97\u4fee\u6539\u4e3a\u4f18\u5316\u8ba1\u7b97\u3002</p>\n<span translate=no>_^_13_^_</span><p>wher<span translate=no>_^_14_^_</span> e \u662f\u6211\u4eec\u5e94\u8be5\u6307\u5b9a\u4e3a\u8d85\u53c2\u6570\u7684\u5185\u5bb9\u3002</p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p>This returns the modified learning rate based on the state. For <em>Adam</em> this is just the specified learning rate for the parameter group, <span translate=no>_^_0_^_</span>.</p>\n": "<h3>\u83b7\u53d6\u5b66\u4e60\u7387</h3>\n<p>\u8fd9\u5c06\u6839\u636e\u72b6\u6001\u8fd4\u56de\u4fee\u6539\u540e\u7684\u5b66\u4e60\u901f\u7387\u3002\u5bf9\u4e8e <em>Adam</em> \u6765\u8bf4\uff0c\u8fd9\u53ea\u662f\u53c2\u6570\u7ec4\u7684\u6307\u5b9a\u5b66\u4e60\u901f\u7387<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>\u521d\u59cb\u5316\u53c2\u6570\u72b6\u6001</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_16_^_</span>.</li></ul>\n": "<h3>\u521d\u59cb\u5316\u4f18\u5316\u5668</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\u5217\u8868</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5b66\u4e60\u7387<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f (<span translate=no>_^_4_^_</span>,<span translate=no>_^_5_^_</span>) \u7684\u5143\u7ec4</li>\n<li><span translate=no>_^_6_^_</span>\u662f<span translate=no>_^_7_^_</span>\u6216<span translate=no>_^_8_^_</span>\u57fa\u4e8e<span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span>\u662f\u5728\u4e2d<span translate=no>_^_11_^_</span>\u5b9a\u4e49\u7684\u7c7b\u7684\u5b9e\u4f8b <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li><span translate=no>_^_13_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u662f\u5426\u5728\u6dfb\u52a0\u540e\u901a\u8fc7\u8fd9\u6837\u505a\u6765\u4f18\u5316\u7b2c\u4e8c\u4e2a\u65f6\u523b\u7684\u504f\u5dee\u6821\u6b63<span translate=no>_^_14_^_</span></li>\n<li><span translate=no>_^_15_^_</span>\u662f\u7ec4\u503c\u7684\u9ed8\u8ba4\u5b57\u5178\u3002\u5f53\u4f60\u60f3\u6269\u5c55\u7c7b\u65f6\uff0c\u8fd9\u5f88\u6709\u7528<span translate=no>_^_16_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u5bf9\u7ed9\u5b9a\u53c2\u6570\u5f20\u91cf\u6267\u884c\u66f4\u65b0\u6b65\u9aa4</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u7684\u5f53\u524d\u68af<span translate=no>_^_3_^_</span>\u5ea6\u5f20\u91cf<span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Bias correction term for <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> </p>\n": "<p>\u504f\u5dee\u6821\u6b63\u672f\u8bed<span translate=no>_^_0_^_</span>\uff0c<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u8ba1\u7b97\u4f53\u91cd\u8870\u51cf</p>\n",
|
||||
"<p>Computation without optimization </p>\n": "<p>\u65e0\u9700\u4f18\u5316\u7684\u8ba1\u7b97</p>\n",
|
||||
"<p>Exponential moving average of gradients, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u68af\u5ea6\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Exponential moving average of squared gradient values, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u68af\u5ea6\u5e73\u65b9\u503c\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u83b7\u53d6<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get learning rate </p>\n": "<p>\u83b7\u53d6\u5b66\u4e60\u7387</p>\n",
|
||||
"<p>In-place calculation of <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5c31\u5730\u8ba1\u7b97<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p><span translate=no>_^_0_^_</span>\u589e\u52a0\u4f18\u5316\u5668\u6b65\u6570</p>\n",
|
||||
"<p>Perform <em>Adam</em> update </p>\n": "<p>\u6267\u884c <em>Adam</em> \u66f4\u65b0</p>\n",
|
||||
"<p>This is the number of optimizer steps taken on the parameter, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8fd9\u662f\u4f18\u5316\u5668\u5bf9\u53c2\u6570\u91c7\u53d6\u7684\u6b65\u9aa4\u6570\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Whether to optimize the computation </p>\n": "<p>\u662f\u5426\u4f18\u5316\u8ba1\u7b97</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer": "Adam \u4f18\u5316\u5668\u7684\u4e00\u4e2a\u7b80\u5355\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b",
|
||||
"Adam Optimizer": "\u4e9a\u5f53\u4f18\u5316\u5668"
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"<h1>Adam Optimizer for Half Precision Training</h1>\n": "<h1>\u534a\u7cbe\u5ea6\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7528\u306e Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h1>\n",
|
||||
"<h2>Adam Optimizer for Half Precision Training</h2>\n<p>We extend <a href=\"adam.html\">Adam Optimizer</a> but use FP32 to store gradients and moments.</p>\n": "<h2>\u534a\u7cbe\u5ea6\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7528\u306e Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h2>\n<p><a href=\"adam.html\">Adam Optimizer\u3092\u62e1\u5f35\u3057\u307e\u3057\u305f\u304c</a>\u3001\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3068\u30e2\u30fc\u30e1\u30f3\u30c8\u306e\u4fdd\u5b58\u306b\u306fFP32\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<h2>Gradient Scaler with half precision gradients</h2>\n<p>We extend PyTorch gradient scaler to use FP32 gradients.</p>\n": "<h2>\u534a\u7cbe\u5ea6\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30b9\u30b1\u30fc\u30e9\u30fc</h2>\n<p>PyTorch \u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30b9\u30b1\u30fc\u30e9\u30fc\u3092 FP32 \u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u4f7f\u7528\u3059\u308b\u3088\u3046\u306b\u62e1\u5f35\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n<p>All the state tensors use FP32.</p>\n": "<h3>\u30d1\u30e9\u30e1\u30fc\u30bf\u72b6\u614b\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_3_^_</span></li></ul>\n<p>\u3059\u3079\u3066\u306e\u30b9\u30c6\u30fc\u30c8\u30c6\u30f3\u30bd\u30eb\u306f FP32 \u3092\u4f7f\u7528\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u4e0e\u3048\u3089\u308c\u305f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb\u306e\u66f4\u65b0\u30b9\u30c6\u30c3\u30d7\u3092\u5b9f\u884c\u3059\u308b</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u73fe\u5728\u306e\u52fe\u914d\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u4f53\u91cd\u6e1b\u5c11\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Call the <a href=\"adam.html\">Adam Optimizer</a> initializer </p>\n": "<p><a href=\"adam.html\">Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u30a4\u30cb\u30b7\u30e3\u30e9\u30a4\u30b6\u30fc\u3092\u547c\u3073\u51fa\u3059</a></p>\n",
|
||||
"<p>Exponential moving average of gradients, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u52fe\u914d\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3001<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Exponential moving average of squared gradient values, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4e8c\u4e57\u52fe\u914d\u5024\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3001<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u53d6\u5f97\u3057\u3066 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get the FP32 gradients if available </p>\n": "<p>\u53ef\u80fd\u306a\u5834\u5408\u306f FP32 \u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the FP32 parameters </p>\n": "<p>FP32 \u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>If we are using the <span translate=no>_^_0_^_</span> optimizer set <span translate=no>_^_1_^_</span> to the FP32 gradients </p>\n": "<p>FP32 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306b\u8a2d\u5b9a\u3055\u308c\u305f\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f7f\u7528\u3057\u3066\u3044\u308b\u5834\u5408</p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p><span translate=no>_^_0_^_</span>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u30b9\u30c6\u30c3\u30d7\u6570\u3092\u5897\u3084\u3059</p>\n",
|
||||
"<p>Loop through parameters </p>\n": "<p>\u30eb\u30fc\u30d7\u30b9\u30eb\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf</p>\n",
|
||||
"<p>Maintain a FP32 copy of the parameters </p>\n": "<p>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e FP32 \u30b3\u30d4\u30fc\u3092\u7ba1\u7406</p>\n",
|
||||
"<p>Not implemented for sparse tensors </p>\n": "<p>\u30b9\u30d1\u30b9\u30c6\u30f3\u30bd\u30eb\u306b\u306f\u5b9f\u88c5\u3055\u308c\u3066\u3044\u307e\u305b\u3093</p>\n",
|
||||
"<p>Otherwise, convert the gradients to FP32 </p>\n": "<p>\u305d\u308c\u4ee5\u5916\u306e\u5834\u5408\u306f\u3001\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092 FP32 \u306b\u5909\u63db\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Otherwise, do not convert the gradients to FP32 </p>\n": "<p>\u305d\u308c\u4ee5\u5916\u306e\u5834\u5408\u306f\u3001\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092 FP32 \u306b\u5909\u63db\u3057\u306a\u3044\u3067\u304f\u3060\u3055\u3044\u3002</p>\n",
|
||||
"<p>Parameter to store 32 bit gradients. This get populated by the <span translate=no>_^_0_^_</span> defined below. </p>\n": "<p>32 \u30d3\u30c3\u30c8\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u683c\u7d0d\u3059\u308b\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3002<span translate=no>_^_0_^_</span>\u3053\u308c\u306b\u306f\u4ee5\u4e0b\u306e\u5b9a\u7fa9\u304c\u5165\u529b\u3055\u308c\u307e\u3059</p>\u3002\n",
|
||||
"<p>Perform <em>Adam</em> update </p>\n": "<p><em>Adam</em> \u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set the parameters </p>\n": "<p>\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Skip non-trainable parameters </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e0d\u53ef\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u30b9\u30ad\u30c3\u30d7</p>\n",
|
||||
"<p>This is the number of optimizer steps taken on the parameter, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3053\u308c\u306f\u3001\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u306b\u5bfe\u3057\u3066\u5b9f\u884c\u3055\u308c\u305f\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u30b9\u30c6\u30c3\u30d7\u306e\u6570\u3067\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Unscale all the gradients </p>\n": "<p>\u3059\u3079\u3066\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30b9\u30b1\u30fc\u30eb\u89e3\u9664</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer": "Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u7c21\u5358\u306a PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb",
|
||||
"Adam Optimizer for Half Precision Training": "\u534a\u7cbe\u5ea6\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7528\u306e Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"<h1>Adam Optimizer for Half Precision Training</h1>\n": "<h1>\u0d85\u0da9\u0d9a\u0dca\u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h1>\n",
|
||||
"<h2>Adam Optimizer for Half Precision Training</h2>\n<p>We extend <a href=\"adam.html\">Adam Optimizer</a> but use FP32 to store gradients and moments.</p>\n": "<h2>\u0d85\u0da9\u0d9a\u0dca\u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0d85\u0db4\u0dd2 <a href=\"adam.html\">\u0d87\u0da9\u0db8\u0dca \u0d94\u0db4\u0dca\u0da7\u0dd2\u0db8\u0dba\u0dd2\u0dc3\u0dbb\u0dca</a> \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dbb\u0db1 \u0db1\u0db8\u0dd4\u0dad\u0dca \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2 \u0dc3\u0dc4 \u0db8\u0ddc\u0dc4\u0ddc\u0dad \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf FP32 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
|
||||
"<h2>Gradient Scaler with half precision gradients</h2>\n<p>We extend PyTorch gradient scaler to use FP32 gradients.</p>\n": "<h2>\u0d85\u0dbb\u0dca\u0db0\u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc3\u0dc4\u0dd2\u0dad \u0d9c\u0dca\u0dbb\u0dda\u0da9\u0dd2\u0dba\u0db1\u0dca\u0da7\u0dca \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba</h2>\n<p>FP32\u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0daf\u0dd2\u0d9c\u0dd4 \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n<p>All the state tensors use FP32.</p>\n": "<h3>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dad\u0dad\u0dca\u0dc0\u0dba\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n</ul><li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_3_^_</span></li>\n<p>\u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8\u0dbb\u0dcf\u0da2\u0dca\u0dba \u0d86\u0dad\u0dad\u0dd3\u0db1\u0dca FP32 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u0daf\u0dd3\u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0d9a\u0dca \u0d9c\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n<li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba <span translate=no>_^_3_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca \u0db5\u0dbd\u0dba \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a tensor \u0dc0\u0dda <span translate=no>_^_4_^_</span> </li>\n</ul><li><span translate=no>_^_5_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_6_^_</span></li>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u0db6\u0dbb\u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Call the <a href=\"adam.html\">Adam Optimizer</a> initializer </p>\n": "<p><a href=\"adam.html\">\u0d87\u0da9\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab</a> \u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a\u0dba \u0d85\u0db8\u0dad\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Exponential moving average of gradients, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9a \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Exponential moving average of squared gradient values, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc0\u0dbb\u0dca\u0d9c\u0db5\u0dbd\u0dba \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc0\u0da7\u0dd2\u0db1\u0dcf\u0d9a\u0db8\u0dca \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dbd\u0db6\u0dcf <span translate=no>_^_0_^_</span> \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Get the FP32 gradients if available </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2 \u0db1\u0db8\u0dca FP32 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the FP32 parameters </p>\n": "<p>FP32\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>If we are using the <span translate=no>_^_0_^_</span> optimizer set <span translate=no>_^_1_^_</span> to the FP32 gradients </p>\n": "<p>\u0d85\u0db4\u0dd2FP32 \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda <span translate=no>_^_0_^_</span> \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba <span translate=no>_^_1_^_</span> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dda \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 \u0dc0\u0dd0\u0da9\u0dd2 <span translate=no>_^_0_^_</span> \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Loop through parameters </p>\n": "<p>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca\u0dc4\u0dbb\u0dc4\u0dcf \u0dbd\u0dd6\u0db4\u0dca </p>\n",
|
||||
"<p>Maintain a FP32 copy of the parameters </p>\n": "<p>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca\u0d9c\u0ddaFP32 \u0db4\u0dd2\u0da7\u0db4\u0dad\u0d9a\u0dca \u0db4\u0dc0\u0dad\u0dca\u0dc0\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Not implemented for sparse tensors </p>\n": "<p>\u0dc0\u0dd2\u0dbb\u0dbd\u0d86\u0dad\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0db1\u0ddc\u0dc0\u0dda </p>\n",
|
||||
"<p>Otherwise, convert the gradients to FP32 </p>\n": "<p>\u0d91\u0dc3\u0dda\u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2\u0db1\u0db8\u0dca, \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba FP32 \u0db6\u0dc0\u0da7 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Otherwise, do not convert the gradients to FP32 </p>\n": "<p>\u0d91\u0dc3\u0dda\u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2\u0db1\u0db8\u0dca, \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a FP32 \u0db6\u0dc0\u0da7 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0db1\u0ddc\u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Parameter to store 32 bit gradients. This get populated by the <span translate=no>_^_0_^_</span> defined below. </p>\n": "<p>\u0db6\u0dd2\u0da7\u0dca\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a 32 \u0d9a\u0dca \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba. \u0db4\u0dc4\u0dad <span translate=no>_^_0_^_</span> \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0db8\u0dd9\u0dba \u0da2\u0db1\u0dcf\u0d9a\u0dd3\u0dbb\u0dca\u0dab \u0dc0\u0dda. </p>\n",
|
||||
"<p>Perform <em>Adam</em> update </p>\n": "<p><em>\u0d86\u0daf\u0db8\u0dca</em> \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dd2\u0daf\u0dd4 </p>\n",
|
||||
"<p>Set the parameters </p>\n": "<p>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca\u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Skip non-trainable parameters </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dc5 \u0db1\u0ddc\u0dc4\u0dd0\u0d9a\u0dd2 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0db8\u0d9f \u0dc4\u0dbb\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>This is the number of optimizer steps taken on the parameter, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba\u0db8\u0dad \u0d9c\u0dd9\u0db1 \u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 \u0db8\u0dd9\u0dba\u0dba\u0dd2, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Unscale all the gradients </p>\n": "<p>\u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer": "\u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dcf\u0dbb\u0d9a\u0dba\u0dda \u0dc3\u0dbb\u0dbd \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba",
|
||||
"Adam Optimizer for Half Precision Training": "\u0d85\u0da9\u0d9a\u0dca \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"<h1>Adam Optimizer for Half Precision Training</h1>\n": "<h1>\u534a\u7cbe\u5ea6\u8bad\u7ec3\u7684 Adam Optimizer</h1>\n",
|
||||
"<h2>Adam Optimizer for Half Precision Training</h2>\n<p>We extend <a href=\"adam.html\">Adam Optimizer</a> but use FP32 to store gradients and moments.</p>\n": "<h2>\u534a\u7cbe\u5ea6\u8bad\u7ec3\u7684 Adam Optimizer</h2>\n<p>\u6211\u4eec\u6269\u5c55\u4e86 <a href=\"adam.html\">Adam Optimizer</a>\uff0c\u4f46\u4f7f\u7528 FP32 \u6765\u5b58\u50a8\u6e10\u53d8\u548c\u65f6\u523b\u3002</p>\n",
|
||||
"<h2>Gradient Scaler with half precision gradients</h2>\n<p>We extend PyTorch gradient scaler to use FP32 gradients.</p>\n": "<h2>\u5177\u6709\u534a\u7cbe\u5ea6\u6e10\u53d8\u7684\u6e10\u53d8\u7f29\u653e\u5668</h2>\n<p>\u6211\u4eec\u5c06 PyTorch \u68af\u5ea6\u7f29\u653e\u5668\u6269\u5c55\u4e3a\u4f7f\u7528 FP32 \u6e10\u53d8\u3002</p>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n<p>All the state tensors use FP32.</p>\n": "<h3>\u521d\u59cb\u5316\u53c2\u6570\u72b6\u6001</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_3_^_</span></li></ul>\n<p>\u6240\u6709\u72b6\u6001\u5f20\u91cf\u90fd\u4f7f\u7528 FP32\u3002</p>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u5bf9\u7ed9\u5b9a\u53c2\u6570\u5f20\u91cf\u6267\u884c\u66f4\u65b0\u6b65\u9aa4</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u7684\u5f53\u524d\u68af<span translate=no>_^_3_^_</span>\u5ea6\u5f20\u91cf<span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u8ba1\u7b97\u4f53\u91cd\u8870\u51cf</p>\n",
|
||||
"<p>Call the <a href=\"adam.html\">Adam Optimizer</a> initializer </p>\n": "<p>\u8c03\u7528 <a href=\"adam.html\">Adam \u4f18\u5316\u5668</a>\u521d\u59cb\u5316\u5668</p>\n",
|
||||
"<p>Exponential moving average of gradients, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u68af\u5ea6\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Exponential moving average of squared gradient values, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u68af\u5ea6\u5e73\u65b9\u503c\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u83b7\u53d6<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get the FP32 gradients if available </p>\n": "<p>\u83b7\u53d6 FP32 \u6e10\u53d8\uff08\u5982\u679c\u6709\uff09</p>\n",
|
||||
"<p>Get the FP32 parameters </p>\n": "<p>\u83b7\u53d6 FP32 \u53c2\u6570</p>\n",
|
||||
"<p>If we are using the <span translate=no>_^_0_^_</span> optimizer set <span translate=no>_^_1_^_</span> to the FP32 gradients </p>\n": "<p>\u5982\u679c\u6211\u4eec\u4f7f\u7528\u8bbe\u7f6e\u4e3a<span translate=no>_^_1_^_</span> FP32 \u6e10\u53d8\u7684<span translate=no>_^_0_^_</span>\u4f18\u5316\u5668</p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p><span translate=no>_^_0_^_</span>\u589e\u52a0\u4f18\u5316\u5668\u6b65\u6570</p>\n",
|
||||
"<p>Loop through parameters </p>\n": "<p>\u5faa\u73af\u6d4f\u89c8\u53c2\u6570</p>\n",
|
||||
"<p>Maintain a FP32 copy of the parameters </p>\n": "<p>\u7ef4\u62a4\u53c2\u6570\u7684 FP32 \u526f\u672c</p>\n",
|
||||
"<p>Not implemented for sparse tensors </p>\n": "<p>\u672a\u9488\u5bf9\u7a00\u758f\u5f20\u91cf\u5b9e\u73b0</p>\n",
|
||||
"<p>Otherwise, convert the gradients to FP32 </p>\n": "<p>\u5426\u5219\uff0c\u5c06\u6e10\u53d8\u8f6c\u6362\u4e3a FP32</p>\n",
|
||||
"<p>Otherwise, do not convert the gradients to FP32 </p>\n": "<p>\u5426\u5219\uff0c\u4e0d\u8981\u5c06\u6e10\u53d8\u8f6c\u6362\u4e3a FP32</p>\n",
|
||||
"<p>Parameter to store 32 bit gradients. This get populated by the <span translate=no>_^_0_^_</span> defined below. </p>\n": "<p>\u7528\u4e8e\u5b58\u50a8 32 \u4f4d\u6e10\u53d8\u7684\u53c2\u6570\u3002\u8fd9\u7531\u4e0b\u9762<span translate=no>_^_0_^_</span>\u5b9a\u4e49\u7684\u586b\u5145\u3002</p>\n",
|
||||
"<p>Perform <em>Adam</em> update </p>\n": "<p>\u6267\u884c <em>Adam</em> \u66f4\u65b0</p>\n",
|
||||
"<p>Set the parameters </p>\n": "<p>\u8bbe\u7f6e\u53c2\u6570</p>\n",
|
||||
"<p>Skip non-trainable parameters </p>\n": "<p>\u8df3\u8fc7\u4e0d\u53ef\u8bad\u7ec3\u7684\u53c2\u6570</p>\n",
|
||||
"<p>This is the number of optimizer steps taken on the parameter, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8fd9\u662f\u4f18\u5316\u5668\u5bf9\u53c2\u6570\u91c7\u53d6\u7684\u6b65\u9aa4\u6570\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Unscale all the gradients </p>\n": "<p>\u53d6\u6d88\u7f29\u653e\u6240\u6709\u6e10\u53d8</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer": "Adam \u4f18\u5316\u5668\u7684\u4e00\u4e2a\u7b80\u5355\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b",
|
||||
"Adam Optimizer for Half Precision Training": "\u534a\u7cbe\u5ea6\u8bad\u7ec3\u7684 Adam Optimizer"
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"<h1>Adam Optimizer with Warmup</h1>\n<p>This extends <a href=\"amsgrad.html\">AMSGrad optimizer</a> and adds a warmup stage.</p>\n": "<h1>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6a5f\u80fd\u4ed8\u304d Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h1>\n<p>\u3053\u308c\u306b\u3088\u308a <a href=\"amsgrad.html\">AMSgrad \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u304c\u62e1\u5f35\u3055\u308c</a>\u3001\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30b9\u30c6\u30fc\u30b8\u304c\u8ffd\u52a0\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<h2>Adam Optimizer with Warmup</h2>\n<p>This class extends from AMSGrad optimizer defined in <a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6a5f\u80fd\u4ed8\u304d Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h2>\n<p>\u3053\u306e\u30af\u30e9\u30b9\u306f\u3001\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b AMSGrad \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u62e1\u5f35\u3057\u305f\u3082\u306e\u3067\u3059\u3002<a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u5b66\u7fd2\u7387\u3092\u53d6\u5f97</h3>\n<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u306f\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30b9\u30c6\u30c3\u30d7\u306e\u6570\u3067\u3059\u3002</p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_17_^_</span>.</li></ul>\n": "<h3>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u30ea\u30b9\u30c8\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u5b66\u7fd2\u7387 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>(,) <span translate=no>_^_4_^_</span> \u306e\u30bf\u30d7\u30eb\u3067\u3059 <span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span>\u307e\u305f\u306f\u305d\u308c\u306b\u57fa\u3065\u3044\u3066\u3044\u308b <span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span><span translate=no>_^_11_^_</span>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3067\u3059 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li>'optimized_update'\u306f\u8ffd\u52a0\u5f8c\u306b\u884c\u3046\u3053\u3068\u3067\u30bb\u30ab\u30f3\u30c9\u30e2\u30fc\u30e1\u30f3\u30c8\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3092\u6700\u9069\u5316\u3059\u308b\u304b\u3069\u3046\u304b\u306e\u30d5\u30e9\u30b0\u3067\u3059 <span translate=no>_^_13_^_</span></li>\n<li><span translate=no>_^_14_^_</span>amsGrad\u3092\u4f7f\u7528\u3059\u308b\u304b\u3001\u30d7\u30ec\u30fc\u30f3\u306aAdam\u306b\u30d5\u30a9\u30fc\u30eb\u30d0\u30c3\u30af\u3059\u308b\u304b\u3092\u793a\u3059\u30d5\u30e9\u30b0\u3067\u3059</li>\n<li><span translate=no>_^_15_^_</span>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30b9\u30c6\u30c3\u30d7\u6570</li>\n<li><span translate=no>_^_16_^_</span>\u30b0\u30eb\u30fc\u30d7\u5024\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u8f9e\u66f8\u3067\u3059\u3002\u3053\u308c\u306f\u3001\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3059\u308b\u5834\u5408\u306b\u4fbf\u5229\u3067\u3059<span translate=no>_^_17_^_</span>\u3002</li></ul>\n",
|
||||
"<p>A linearly increasing learning rate from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5b66\u7fd2\u7387\u304c 1 \u304b\u3089 <span translate=no>_^_0_^_</span> 1 \u306b\u76f4\u7dda\u7684\u306b\u5897\u52a0\u3057\u3066\u3044\u308b <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Constant learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4e00\u5b9a\u306e\u5b66\u7fd2\u7387 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>If we are in warmup stage </p>\n": "<p>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6bb5\u968e\u306e\u5834\u5408</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer with warm-up.": "\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6a5f\u80fd\u4ed8\u304d\u306e Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u7c21\u5358\u306a PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"Adam optimizer with warm-up": "\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6a5f\u80fd\u4ed8\u304d Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"<h1>Adam Optimizer with Warmup</h1>\n<p>This extends <a href=\"amsgrad.html\">AMSGrad optimizer</a> and adds a warmup stage.</p>\n": "<h1>\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d87\u0da9\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"amsgrad.html\">AMSGrad \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db4\u0dd4\u0dc5\u0dd4\u0dbd\u0dca \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dbb \u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dd2\u0dba\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<h2>Adam Optimizer with Warmup</h2>\n<p>This class extends from AMSGrad optimizer defined in <a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d87\u0da9\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0db8\u0dd9\u0db8\u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba AMSGrad \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dd9\u0db1\u0dca \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad <a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>. </p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u0d89\u0d9c\u0dd9\u0db1\u0dd3\u0db8-\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1</h3>\n<p><span translate=no>_^_0_^_</span> \u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_17_^_</span>.</li></ul>\n": "<h3>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0dba\u0dd2 </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dba\u0dd2 <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) \u0d9a tuple \u0dc0\u0dda </li>\n<li><span translate=no>_^_6_^_</span> <span translate=no>_^_7_^_</span> \u0dc4\u0ddd \u0db8\u0dad <span translate=no>_^_8_^_</span> \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> <span translate=no>_^_11_^_</span> \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba\u0dda \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0d9a\u0dd2 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update'\u0dba\u0db1\u0dd4 \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d91\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0daf\u0dd9\u0dc0\u0db1 \u0db8\u0ddc\u0dc4\u0ddc\u0dad\u0dda \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf \u0dba\u0db1\u0dca\u0db1 \u0db0\u0da2\u0dba\u0d9a\u0dd2 <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> \u0d86\u0daf\u0db8\u0dca \u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf AMSGrad \u0dc4\u0ddd \u0dc0\u0dd0\u0da7\u0dd3\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd9\u0db1 \u0db0\u0da2\u0dba\u0d9a\u0dd2 </li>\n<li><span translate=no>_^_15_^_</span> \u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_16_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc1\u0db6\u0dca\u0daf \u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dd2. \u0d94\u0db6\u0da7 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dd2\u0da7 \u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dba\u0ddd\u0da2\u0db1\u0dc0\u0dad\u0dca <span translate=no>_^_17_^_</span>\u0dc0\u0dda. </li></ul>\n",
|
||||
"<p>A linearly increasing learning rate from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0dd2\u0da7 <span translate=no>_^_0_^_</span> \u0dbb\u0dda\u0d9b\u0dd3\u0dba\u0dc0 \u0dc0\u0dd0\u0da9\u0dd2\u0dc0\u0db1 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Constant learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dd2\u0dbb\u0db1\u0dca\u0dad\u0dbb\u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>If we are in warmup stage </p>\n": "<p>\u0d85\u0db4\u0dd2\u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dd2\u0dba\u0d9a \u0dc3\u0dd2\u0da7\u0dd3 \u0db1\u0db8\u0dca </p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer with warm-up.": "\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0dda \u0dc3\u0dbb\u0dbd \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba.",
|
||||
"Adam optimizer with warm-up": "\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8 \u0dc3\u0dc4\u0dd2\u0dad \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"<h1>Adam Optimizer with Warmup</h1>\n<p>This extends <a href=\"amsgrad.html\">AMSGrad optimizer</a> and adds a warmup stage.</p>\n": "<h1>Adam Optimizer \u5e26\u70ed\u8eab</h1>\n<p>\u8fd9\u6269\u5c55\u4e86 <a href=\"amsgrad.html\">AmsGrad \u4f18\u5316\u5668</a>\u5e76\u589e\u52a0\u4e86\u9884\u70ed\u9636\u6bb5\u3002</p>\n",
|
||||
"<h2>Adam Optimizer with Warmup</h2>\n<p>This class extends from AMSGrad optimizer defined in <a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>Adam Optimizer \u5e26\u70ed\u8eab</h2>\n<p>\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 AmsGrad \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684<a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>\u3002</p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u83b7\u53d6\u5b66\u4e60\u7387</h3>\n<p><span translate=no>_^_0_^_</span>\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u9884\u70ed\u6b65\u9aa4\u7684\u6570\u91cf\u3002</p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_17_^_</span>.</li></ul>\n": "<h3>\u521d\u59cb\u5316\u4f18\u5316\u5668</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\u5217\u8868</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5b66\u4e60\u7387<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f (<span translate=no>_^_4_^_</span>,<span translate=no>_^_5_^_</span>) \u7684\u5143\u7ec4</li>\n<li><span translate=no>_^_6_^_</span>\u662f<span translate=no>_^_7_^_</span>\u6216<span translate=no>_^_8_^_</span>\u57fa\u4e8e<span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span>\u662f\u5728\u4e2d<span translate=no>_^_11_^_</span>\u5b9a\u4e49\u7684\u7c7b\u7684\u5b9e\u4f8b <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li>\u201coptimized_update\u201d \u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u5728\u6dfb\u52a0\u540e\u662f\u5426\u8981\u4f18\u5316\u7b2c\u4e8c\u4e2a\u65f6\u523b\u7684\u504f\u5dee\u6821\u6b63<span translate=no>_^_13_^_</span></li>\n<li><span translate=no>_^_14_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u6307\u793a\u662f\u4f7f\u7528 AmsGrad \u8fd8\u662f\u56de\u9000\u5230\u666e\u901a\u7684 Adam</li>\n<li><span translate=no>_^_15_^_</span>\u9884\u70ed\u6b65\u6570</li>\n<li><span translate=no>_^_16_^_</span>\u662f\u7ec4\u503c\u7684\u9ed8\u8ba4\u5b57\u5178\u3002\u5f53\u4f60\u60f3\u6269\u5c55\u7c7b\u65f6\uff0c\u8fd9\u5f88\u6709\u7528<span translate=no>_^_17_^_</span>\u3002</li></ul>\n",
|
||||
"<p>A linearly increasing learning rate from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5b66\u4e60\u7387\u4ece\u7ebf\u6027\u589e\u52a0<span translate=no>_^_0_^_</span>\u5230<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Constant learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6301\u7eed\u7684\u5b66\u4e60\u901f\u7387<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>If we are in warmup stage </p>\n": "<p>\u5982\u679c\u6211\u4eec\u5904\u4e8e\u70ed\u8eab\u9636\u6bb5</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Adam optimizer with warm-up.": "\u4e00\u4e2a\u7b80\u5355\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u7684 Adam \u4f18\u5316\u5668\uff0c\u5e26\u6709\u9884\u70ed\u529f\u80fd\u3002",
|
||||
"Adam optimizer with warm-up": "\u5e26\u70ed\u8eab\u529f\u80fd\u7684 Adam \u4f18\u5316\u5668"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>Adam Optimizer with Warmup and Cosine Decay</h1>\n<p>This extends <a href=\"adam.html\">AMSGrad optimizer</a> and adds a warmup stage.</p>\n": "<h1>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u3068\u30b3\u30b5\u30a4\u30f3\u30c7\u30a3\u30b1\u30a4\u3092\u5099\u3048\u305f Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h1>\n<p>\u3053\u308c\u306b\u3088\u308a <a href=\"adam.html\">AMSgrad \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u304c\u62e1\u5f35\u3055\u308c</a>\u3001\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30b9\u30c6\u30fc\u30b8\u304c\u8ffd\u52a0\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u5b66\u7fd2\u7387\u3092\u53d6\u5f97</h3>\n<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u306f\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30b9\u30c6\u30c3\u30d7\u306e\u6570\u3067\u3059\u3002</p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> total number of steps. Cosine decay reaches 0 at this, but stays at 10% of <span translate=no>_^_17_^_</span> because we take <span translate=no>_^_18_^_</span> </li>\n<li><span translate=no>_^_19_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_20_^_</span>.</li></ul>\n": "<h3>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u30ea\u30b9\u30c8\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u5b66\u7fd2\u7387 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>(,) <span translate=no>_^_4_^_</span> \u306e\u30bf\u30d7\u30eb\u3067\u3059 <span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span>\u307e\u305f\u306f\u305d\u308c\u306b\u57fa\u3065\u3044\u3066\u3044\u308b <span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span><span translate=no>_^_11_^_</span>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3067\u3059 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li>'optimized_update'\u306f\u8ffd\u52a0\u5f8c\u306b\u884c\u3046\u3053\u3068\u3067\u30bb\u30ab\u30f3\u30c9\u30e2\u30fc\u30e1\u30f3\u30c8\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3092\u6700\u9069\u5316\u3059\u308b\u304b\u3069\u3046\u304b\u306e\u30d5\u30e9\u30b0\u3067\u3059 <span translate=no>_^_13_^_</span></li>\n<li><span translate=no>_^_14_^_</span>amsGrad\u3092\u4f7f\u7528\u3059\u308b\u304b\u3001\u30d7\u30ec\u30fc\u30f3\u306aAdam\u306b\u30d5\u30a9\u30fc\u30eb\u30d0\u30c3\u30af\u3059\u308b\u304b\u3092\u793a\u3059\u30d5\u30e9\u30b0\u3067\u3059</li>\n<li><span translate=no>_^_15_^_</span>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30b9\u30c6\u30c3\u30d7\u6570</li>\n<li><span translate=no>_^_16_^_</span>\u30b9\u30c6\u30c3\u30d7\u306e\u7dcf\u6570\u3002\u3053\u306e\u6642\u70b9\u3067\u30b3\u30b5\u30a4\u30f3\u6e1b\u8870\u306f0\u306b\u9054\u3057\u307e\u3059\u304c\u3001<span translate=no>_^_17_^_</span>\u53d6\u308b\u305f\u308110\uff05\u306b\u3068\u3069\u307e\u308a\u307e\u3059 <span translate=no>_^_18_^_</span></li>\n<li><span translate=no>_^_19_^_</span>\u30b0\u30eb\u30fc\u30d7\u5024\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u8f9e\u66f8\u3067\u3059\u3002\u3053\u308c\u306f\u3001\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3059\u308b\u5834\u5408\u306b\u4fbf\u5229\u3067\u3059<span translate=no>_^_20_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Plot learning rate for different warmups and model sizes</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>\u3055\u307e\u3056\u307e\u306a\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u3068\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba\u306e\u5b66\u7fd2\u7387\u3092\u30d7\u30ed\u30c3\u30c8</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p> <a id=\"EmbeddingsWithPositionalEncoding\"></a></p>\n<h2>Adam Optimizer with Warmup and Cosine Decay</h2>\n<p>This class extends from AMSGrad optimizer defined in <a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<p><a id=\"EmbeddingsWithPositionalEncoding\"></a></p>\n<h2>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u3068\u30b3\u30b5\u30a4\u30f3\u30c7\u30a3\u30b1\u30a4\u3092\u5099\u3048\u305f Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h2>\n<p>\u3053\u306e\u30af\u30e9\u30b9\u306f\u3001\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b AMSGrad \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u62e1\u5f35\u3057\u305f\u3082\u306e\u3067\u3059\u3002<a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<p>A linearly increasing learning rate from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5b66\u7fd2\u7387\u304c 1 \u304b\u3089 <span translate=no>_^_0_^_</span> 1 \u306b\u76f4\u7dda\u7684\u306b\u5897\u52a0\u3057\u3066\u3044\u308b <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Constant learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4e00\u5b9a\u306e\u5b66\u7fd2\u7387 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>If we are in warmup stage </p>\n": "<p>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6bb5\u968e\u306e\u5834\u5408</p>\n",
|
||||
"A PyTorch implementation/tutorial of Adam optimizer with warm-up and cosine decay for GPT.": "GPT \u306e\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u3068\u30b3\u30b5\u30a4\u30f3\u30c7\u30a3\u30b1\u30a4\u3092\u5099\u3048\u305f Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"Adam optimizer with warm-up and cosine decay": "\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u3068\u30b3\u30b5\u30a4\u30f3\u30c7\u30a3\u30b1\u30a4\u3092\u5099\u3048\u305f Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>Adam Optimizer with Warmup and Cosine Decay</h1>\n<p>This extends <a href=\"adam.html\">AMSGrad optimizer</a> and adds a warmup stage.</p>\n": "<h1>\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca\u0dc4\u0dcf \u0d9a\u0ddc\u0dc3\u0dd3\u0db1\u0dca \u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d87\u0da9\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"adam.html\">AMSGrad \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db4\u0dd4\u0dc5\u0dd4\u0dbd\u0dca \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dbb \u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dd2\u0dba\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u0d89\u0d9c\u0dd9\u0db1\u0dd3\u0db8-\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1</h3>\n<p><span translate=no>_^_0_^_</span> \u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> total number of steps. Cosine decay reaches 0 at this, but stays at 10% of <span translate=no>_^_17_^_</span> because we take <span translate=no>_^_18_^_</span> </li>\n<li><span translate=no>_^_19_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_20_^_</span>.</li></ul>\n": "<h3>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0dba\u0dd2 </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dba\u0dd2 <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) \u0d9a tuple \u0dc0\u0dda </li>\n<li><span translate=no>_^_6_^_</span> <span translate=no>_^_7_^_</span> \u0dc4\u0ddd \u0db8\u0dad <span translate=no>_^_8_^_</span> \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> <span translate=no>_^_11_^_</span> \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba\u0dda \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0d9a\u0dd2 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update'\u0dba\u0db1\u0dd4 \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d91\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0daf\u0dd9\u0dc0\u0db1 \u0db8\u0ddc\u0dc4\u0ddc\u0dad\u0dda \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf \u0dba\u0db1\u0dca\u0db1 \u0db0\u0da2\u0dba\u0d9a\u0dd2 <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> \u0d86\u0daf\u0db8\u0dca \u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf AMSGrad \u0dc4\u0ddd \u0dc0\u0dd0\u0da7\u0dd3\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd9\u0db1 \u0db0\u0da2\u0dba\u0d9a\u0dd2 </li>\n<li><span translate=no>_^_15_^_</span> \u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_16_^_</span> \u0db8\u0dd4\u0dc5\u0dd4 \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1. \u0d9a\u0ddc\u0dc3\u0dba\u0dd2\u0db1\u0dca \u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0db8\u0dda \u0dc0\u0db1 \u0dc0\u0dd2\u0da7 0 \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0dc5\u0d9f\u0dcf \u0dc0\u0dda, \u0db1\u0db8\u0dd4\u0dad\u0dca \u0d85\u0db4 \u0d9c\u0db1\u0dca\u0db1\u0dcf <span translate=no>_^_17_^_</span> \u0db1\u0dd2\u0dc3\u0dcf 10% \u0d9a \u0dbb\u0dd0\u0db3\u0dd3 \u0dc3\u0dd2\u0da7\u0dd2\u0dba\u0dd2 <span translate=no>_^_18_^_</span> </li>\n<li><span translate=no>_^_19_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc1\u0db6\u0dca\u0daf \u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dd2. \u0d94\u0db6\u0da7 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dd2\u0da7 \u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dba\u0ddd\u0da2\u0db1\u0dc0\u0dad\u0dca <span translate=no>_^_20_^_</span>\u0dc0\u0dda. </li></ul>\n",
|
||||
"<h3>Plot learning rate for different warmups and model sizes</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>\u0dc0\u0dd2\u0dc0\u0dd2\u0db0\u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0dc3\u0dc4 \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dca\u0dbd\u0ddc\u0da7\u0dca \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p> <a id=\"EmbeddingsWithPositionalEncoding\"></a></p>\n<h2>Adam Optimizer with Warmup and Cosine Decay</h2>\n<p>This class extends from AMSGrad optimizer defined in <a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<p> <a id=\"EmbeddingsWithPositionalEncoding\"></a></p>\n<h2>\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca\u0dc4\u0dcf \u0d9a\u0ddc\u0dc3\u0dd3\u0db1\u0dca \u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d87\u0da9\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0db8\u0dd9\u0db8\u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba AMSGrad \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dd9\u0db1\u0dca \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad <a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>. </p>\n",
|
||||
"<p>A linearly increasing learning rate from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0dd2\u0da7 <span translate=no>_^_0_^_</span> \u0dbb\u0dda\u0d9b\u0dd3\u0dba\u0dc0 \u0dc0\u0dd0\u0da9\u0dd2\u0dc0\u0db1 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Constant learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dd2\u0dbb\u0db1\u0dca\u0dad\u0dbb\u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>If we are in warmup stage </p>\n": "<p>\u0d85\u0db4\u0dd2\u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dd2\u0dba\u0d9a \u0dc3\u0dd2\u0da7\u0dd3 \u0db1\u0db8\u0dca </p>\n",
|
||||
"A PyTorch implementation/tutorial of Adam optimizer with warm-up and cosine decay for GPT.": "GPT \u0dc3\u0db3\u0dc4\u0dcf \u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0dc4\u0dcf \u0d9a\u0ddc\u0dc3\u0dba\u0dd2\u0db1\u0dca \u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0dc3\u0dc4\u0dd2\u0dad \u0d87\u0da9\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0dda \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba.",
|
||||
"Adam optimizer with warm-up and cosine decay": "\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0dc4\u0dcf \u0d9a\u0ddc\u0dc3\u0dba\u0dd2\u0db1\u0dca \u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0dc3\u0dc4\u0dd2\u0dad \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>Adam Optimizer with Warmup and Cosine Decay</h1>\n<p>This extends <a href=\"adam.html\">AMSGrad optimizer</a> and adds a warmup stage.</p>\n": "<h1>\u5e26\u6709\u9884\u70ed\u548c\u4f59\u5f26\u8870\u51cf\u7684 Adam Optimizer</h1>\n<p>\u8fd9\u6269\u5c55\u4e86 <a href=\"adam.html\">AmsGrad \u4f18\u5316\u5668</a>\u5e76\u589e\u52a0\u4e86\u9884\u70ed\u9636\u6bb5\u3002</p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u83b7\u53d6\u5b66\u4e60\u7387</h3>\n<p><span translate=no>_^_0_^_</span>\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u9884\u70ed\u6b65\u9aa4\u7684\u6570\u91cf\u3002</p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> total number of steps. Cosine decay reaches 0 at this, but stays at 10% of <span translate=no>_^_17_^_</span> because we take <span translate=no>_^_18_^_</span> </li>\n<li><span translate=no>_^_19_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_20_^_</span>.</li></ul>\n": "<h3>\u521d\u59cb\u5316\u4f18\u5316\u5668</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\u5217\u8868</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5b66\u4e60\u7387<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f (<span translate=no>_^_4_^_</span>,<span translate=no>_^_5_^_</span>) \u7684\u5143\u7ec4</li>\n<li><span translate=no>_^_6_^_</span>\u662f<span translate=no>_^_7_^_</span>\u6216<span translate=no>_^_8_^_</span>\u57fa\u4e8e<span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span>\u662f\u5728\u4e2d<span translate=no>_^_11_^_</span>\u5b9a\u4e49\u7684\u7c7b\u7684\u5b9e\u4f8b <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li>\u201coptimized_update\u201d \u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u5728\u6dfb\u52a0\u540e\u662f\u5426\u8981\u4f18\u5316\u7b2c\u4e8c\u4e2a\u65f6\u523b\u7684\u504f\u5dee\u6821\u6b63<span translate=no>_^_13_^_</span></li>\n<li><span translate=no>_^_14_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u6307\u793a\u662f\u4f7f\u7528 AmsGrad \u8fd8\u662f\u56de\u9000\u5230\u666e\u901a\u7684 Adam</li>\n<li><span translate=no>_^_15_^_</span>\u9884\u70ed\u6b65\u6570</li>\n<li><span translate=no>_^_16_^_</span>\u603b\u6b65\u6570\u3002\u6b64\u65f6\u4f59\u5f26\u8870\u51cf\u8fbe\u5230 0\uff0c\u4f46\u4fdd\u6301\u5728 10%\uff0c<span translate=no>_^_17_^_</span>\u56e0\u4e3a\u6211\u4eec\u5f97</li>\u5230<span translate=no>_^_18_^_</span>\n<li><span translate=no>_^_19_^_</span>\u662f\u7ec4\u503c\u7684\u9ed8\u8ba4\u5b57\u5178\u3002\u5f53\u4f60\u60f3\u6269\u5c55\u7c7b\u65f6\uff0c\u8fd9\u5f88\u6709\u7528<span translate=no>_^_20_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Plot learning rate for different warmups and model sizes</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>\u7ed8\u5236\u4e0d\u540c\u9884\u70ed\u548c\u6a21\u578b\u5927\u5c0f\u7684\u5b66\u4e60\u901f\u7387</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p> <a id=\"EmbeddingsWithPositionalEncoding\"></a></p>\n<h2>Adam Optimizer with Warmup and Cosine Decay</h2>\n<p>This class extends from AMSGrad optimizer defined in <a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<p><a id=\"EmbeddingsWithPositionalEncoding\"></a></p>\n<h2>\u5e26\u6709\u9884\u70ed\u548c\u4f59\u5f26\u8870\u51cf\u7684 Adam Optimizer</h2>\n<p>\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 AmsGrad \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684<a href=\"amsgrad.html\"><span translate=no>_^_0_^_</span></a>\u3002</p>\n",
|
||||
"<p>A linearly increasing learning rate from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5b66\u4e60\u7387\u4ece\u7ebf\u6027\u589e\u52a0<span translate=no>_^_0_^_</span>\u5230<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Constant learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6301\u7eed\u7684\u5b66\u4e60\u901f\u7387<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>If we are in warmup stage </p>\n": "<p>\u5982\u679c\u6211\u4eec\u5904\u4e8e\u70ed\u8eab\u9636\u6bb5</p>\n",
|
||||
"A PyTorch implementation/tutorial of Adam optimizer with warm-up and cosine decay for GPT.": "Adam \u4f18\u5316\u5668\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\uff0c\u5177\u6709 GPT \u7684\u9884\u70ed\u548c\u4f59\u5f26\u8870\u51cf\u3002",
|
||||
"Adam optimizer with warm-up and cosine decay": "\u5177\u6709\u9884\u70ed\u548c\u4f59\u5f26\u8870\u51cf\u7684 Adam \u4f18\u5316\u5668"
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"<h1>AMSGrad</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/1904.09237\">On the Convergence of Adam and Beyond</a>.</p>\n<p>We implement this as an extension to our <a href=\"adam.html\">Adam optimizer implementation</a>. The implementation it self is really small since it's very similar to Adam.</p>\n<p>We also have an implementation of the synthetic example described in the paper where Adam fails to converge.</p>\n": "<h1>\u30de\u30b9\u30b0\u30e9\u30fc\u30c9</h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/1904.09237\">\u30a2\u30c0\u30e0\u306e\u53ce\u675f\u3068\u5f7c\u65b9\u300d<a href=\"https://pytorch.org\">\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n<p>\u3053\u308c\u3092 <a href=\"adam.html\">Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u5b9f\u88c5\u306e\u62e1\u5f35\u3068\u3057\u3066\u5b9f\u88c5\u3057\u307e\u3059</a>\u3002Adam\u3068\u975e\u5e38\u306b\u4f3c\u3066\u3044\u308b\u306e\u3067\u3001\u5b9f\u88c5\u81ea\u4f53\u306f\u975e\u5e38\u306b\u5c0f\u3055\u3044\u3067\u3059\u3002</p>\n<p>\u307e\u305f\u3001\u8ad6\u6587\u3067\u8aac\u660e\u3057\u305fAdam\u304c\u53ce\u675f\u3057\u306a\u3044\u5408\u6210\u4f8b\u306e\u5b9f\u88c5\u3082\u3042\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<h2>AMSGrad Optimizer</h2>\n<p>This class extends from Adam optimizer defined in <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>. Adam optimizer is extending the class <span translate=no>_^_1_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_2_^_</span></a>.</p>\n": "<h2>\u30de\u30b9\u30b0\u30e9\u30fc\u30c9\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h2>\n<p>\u3053\u306e\u30af\u30e9\u30b9\u306f\u3001\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u62e1\u5f35\u3057\u305f\u3082\u306e\u3067\u3059\u3002<a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>Adam <span translate=no>_^_1_^_</span> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u306f\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3057\u3066\u3044\u307e\u3059</p>\u3002<a href=\"index.html\"><span translate=no>_^_2_^_</span></a>\n",
|
||||
"<h2>Synthetic Experiment</h2>\n<p>This is the synthetic experiment described in the paper, that shows a scenario where <em>Adam</em> fails.</p>\n<p>The paper (and Adam) formulates the problem of optimizing as minimizing the expected value of a function, <span translate=no>_^_0_^_</span> with respect to the parameters <span translate=no>_^_1_^_</span>. In the stochastic training setting we do not get hold of the function <span translate=no>_^_2_^_</span> it self; that is, when you are optimizing a NN <span translate=no>_^_3_^_</span> would be the function on entire batch of data. What we actually evaluate is a mini-batch so the actual function is realization of the stochastic <span translate=no>_^_4_^_</span>. This is why we are talking about an expected value. So let the function realizations be <span translate=no>_^_5_^_</span> for each time step of training.</p>\n<p>We measure the performance of the optimizer as the regret, <span translate=no>_^_6_^_</span> where <span translate=no>_^_7_^_</span> is the parameters at time step <span translate=no>_^_8_^_</span>, and <span translate=no>_^_9_^_</span> is the optimal parameters that minimize <span translate=no>_^_10_^_</span>.</p>\n<p>Now lets define the synthetic problem,</p>\n<span translate=no>_^_11_^_</span><p>where <span translate=no>_^_12_^_</span>. The optimal solution is <span translate=no>_^_13_^_</span>.</p>\n<p>This code will try running <em>Adam</em> and <em>AMSGrad</em> on this problem.</p>\n": "<h2>\u5408\u6210\u5b9f\u9a13</h2>\n<p>\u3053\u308c\u306f\u8ad6\u6587\u3067\u8aac\u660e\u3055\u308c\u3066\u3044\u308b\u5408\u6210\u5b9f\u9a13\u3067\u3001<em>\u30a2\u30c0\u30e0\u304c\u5931\u6557\u3059\u308b\u30b7\u30ca\u30ea\u30aa\u3092\u793a\u3057\u3066\u3044\u307e\u3059</em>\u3002</p>\n<p>\u8ad6\u6587\uff08\u3068\u30a2\u30c0\u30e0\uff09\u306f\u3001\u6700\u9069\u5316\u306e\u554f\u984c\u3092\u3001\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u306b\u95a2\u3059\u308b\u95a2\u6570\u306e\u671f\u5f85\u5024\u3092\u6700\u5c0f\u5316\u3059\u308b\u3053\u3068\u3068\u3057\u3066\u5b9a\u5f0f\u5316\u3057\u3066\u3044\u307e\u3059\u3002<span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span>\u78ba\u7387\u7684\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u8a2d\u5b9a\u3067\u306f\u3001<span translate=no>_^_2_^_</span>\u95a2\u6570\u81ea\u4f53\u3092\u628a\u63e1\u3059\u308b\u3053\u3068\u306f\u3067\u304d\u307e\u305b\u3093\u3002\u3064\u307e\u308a\u3001\u6700\u9069\u5316\u3059\u308b\u3068\u3001NN <span translate=no>_^_3_^_</span> \u306f\u30c7\u30fc\u30bf\u306e\u30d0\u30c3\u30c1\u5168\u4f53\u306b\u5bfe\u3059\u308b\u95a2\u6570\u306b\u306a\u308a\u307e\u3059\u3002\u5b9f\u969b\u306b\u8a55\u4fa1\u3059\u308b\u306e\u306f\u30df\u30cb\u30d0\u30c3\u30c1\u306a\u306e\u3067\u3001\u5b9f\u969b\u306e\u95a2\u6570\u306f\u78ba\u7387\u8ad6\u306e\u5b9f\u73fe\u3067\u3059\u3002<span translate=no>_^_4_^_</span>\u3053\u308c\u304c\u671f\u5f85\u5024\u306b\u3064\u3044\u3066\u8a71\u3057\u3066\u3044\u308b\u7406\u7531\u3067\u3059\u3002\u305d\u3053\u3067\u3001<span translate=no>_^_5_^_</span>\u6a5f\u80fd\u306e\u5b9f\u73fe\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u5404\u30bf\u30a4\u30e0\u30b9\u30c6\u30c3\u30d7\u3067\u884c\u3046\u3068\u3057\u307e\u3057\u3087\u3046</p>\u3002\n<p>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u306e\u6027\u80fd\u3092\u5f8c\u6094\u3068\u3057\u3066\u6e2c\u5b9a\u3057\u307e\u3059\u3002<span translate=no>_^_6_^_</span>\u3053\u3053\u3067\u3001<span translate=no>_^_7_^_</span>\u306f\u30bf\u30a4\u30e0\u30b9\u30c6\u30c3\u30d7\u3067\u306e\u30d1\u30e9\u30e1\u30fc\u30bf<span translate=no>_^_8_^_</span>\u3001<span translate=no>_^_9_^_</span>\u306f\u6700\u5c0f\u5316\u3059\u308b\u6700\u9069\u306a\u30d1\u30e9\u30e1\u30fc\u30bf\u3067\u3059\u3002<span translate=no>_^_10_^_</span></p>\n<p>\u305d\u308c\u3067\u306f\u3001\u7dcf\u5408\u7684\u306a\u554f\u984c\u3092\u5b9a\u7fa9\u3057\u307e\u3057\u3087\u3046\u3002</p>\n<span translate=no>_^_11_^_</span><p>\u3069\u3053<span translate=no>_^_12_^_</span>\u3002\u6700\u9069\u306a\u89e3\u6c7a\u7b56\u306f\u3067\u3059<span translate=no>_^_13_^_</span>\u3002</p>\n<p>\u3053\u306e\u30b3\u30fc\u30c9\u3067\u306f\u3001\u3053\u306e\u554f\u984c\u306b\u5bfe\u3057\u3066 <em>Adam \u3068 <em>AmsGrad</em></em> \u3092\u5b9f\u884c\u3057\u3066\u307f\u307e\u3059\u3002</p>\n",
|
||||
"<h3><span translate=no>_^_0_^_</span></h3>\n": "<h3><span translate=no>_^_0_^_</span></h3>\n",
|
||||
"<h3>Calculate <span translate=no>_^_0_^_</span> and and <span translate=no>_^_1_^_</span> or <span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_4_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_5_^_</span> is the current gradient tensor <span translate=no>_^_6_^_</span> for the parameter <span translate=no>_^_7_^_</span></li></ul>\n": "<h3><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u8a08\u7b97\u304a\u3088\u3073\u307e\u305f\u306f <span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_4_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_5_^_</span><span translate=no>_^_6_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u73fe\u5728\u306e\u52fe\u914d\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_7_^_</span></li></ul>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>\u30d1\u30e9\u30e1\u30fc\u30bf\u72b6\u614b\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_16_^_</span>.</li></ul>\n": "<h3>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u30ea\u30b9\u30c8\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u5b66\u7fd2\u7387 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>(,) <span translate=no>_^_4_^_</span> \u306e\u30bf\u30d7\u30eb\u3067\u3059 <span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span>\u307e\u305f\u306f\u305d\u308c\u306b\u57fa\u3065\u3044\u3066\u3044\u308b <span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span><span translate=no>_^_11_^_</span>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3067\u3059 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li>'optimized_update'\u306f\u8ffd\u52a0\u5f8c\u306b\u884c\u3046\u3053\u3068\u3067\u30bb\u30ab\u30f3\u30c9\u30e2\u30fc\u30e1\u30f3\u30c8\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3092\u6700\u9069\u5316\u3059\u308b\u304b\u3069\u3046\u304b\u306e\u30d5\u30e9\u30b0\u3067\u3059 <span translate=no>_^_13_^_</span></li>\n<li><span translate=no>_^_14_^_</span>amsGrad\u3092\u4f7f\u7528\u3059\u308b\u304b\u3001\u30d7\u30ec\u30fc\u30f3\u306aAdam\u306b\u30d5\u30a9\u30fc\u30eb\u30d0\u30c3\u30af\u3059\u308b\u304b\u3092\u793a\u3059\u30d5\u30e9\u30b0\u3067\u3059</li>\n<li><span translate=no>_^_15_^_</span>\u30b0\u30eb\u30fc\u30d7\u5024\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u8f9e\u66f8\u3067\u3059\u3002\u3053\u308c\u306f\u3001\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3059\u308b\u5834\u5408\u306b\u4fbf\u5229\u3067\u3059<span translate=no>_^_16_^_</span>\u3002</li></ul>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span>.</p>\n<p>\ud83e\udd14 I feel you should be taking / maintaining the max of the bias corrected second exponential average of squared gradient. But this is how it's <a href=\"https://github.com/pytorch/pytorch/blob/19f4c5110e8bcad5e7e75375194262fca0a6293a/torch/optim/functional.py#L90\">implemented in PyTorch also</a>. I guess it doesn't really matter since bias correction only increases the value and it only makes an actual difference during the early few steps of the training. </p>\n": "<p>\u8a08\u7b97<span translate=no>_^_0_^_</span>\u3002</p>\n<p>\ud83e\udd14 \u4e8c\u4e57\u52fe\u914d\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u5f8c\u306e\u7b2c\u4e8c\u6307\u6570\u5e73\u5747\u306e\u6700\u5927\u5024\u3092\u3068\u308b/\u7dad\u6301\u3059\u3079\u304d\u3060\u3068\u601d\u3044\u307e\u3059\u3002\u3057\u304b\u3057\u3001<a href=\"https://github.com/pytorch/pytorch/blob/19f4c5110e8bcad5e7e75375194262fca0a6293a/torch/optim/functional.py#L90\">PyTorch\u3067\u3082\u3053\u306e\u3088\u3046\u306b\u5b9f\u88c5\u3055\u308c\u3066\u3044\u307e\u3059</a>\u3002\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u306f\u5024\u3092\u5897\u3084\u3059\u3060\u3051\u3067\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u521d\u671f\u306e\u6570\u30b9\u30c6\u30c3\u30d7\u3067\u5b9f\u969b\u306b\u9055\u3044\u304c\u51fa\u308b\u3060\u3051\u306a\u306e\u3067\u3001\u305d\u308c\u307b\u3069\u91cd\u8981\u3067\u306f\u306a\u3044\u3068\u601d\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u52fe\u914d\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Call <span translate=no>_^_0_^_</span> of Adam optimizer which we are extending </p>\n": "<p><span translate=no>_^_0_^_</span>\u62e1\u5f35\u4e2d\u306eCall of Adam\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</p>\n",
|
||||
"<p>Clear gradients </p>\n": "<p>\u30af\u30ea\u30a2\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Create experiment to record results </p>\n": "<p>\u30c6\u30b9\u30c8\u3092\u4f5c\u6210\u3057\u3066\u7d50\u679c\u3092\u8a18\u9332\u3059\u308b</p>\n",
|
||||
"<p>Define <span translate=no>_^_0_^_</span> parameter </p>\n": "<p><span translate=no>_^_0_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u5b9a\u7fa9</p>\n",
|
||||
"<p>Fall back to <em>Adam</em> if the parameter group is not using <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u304c\u4f7f\u7528\u3057\u3066\u3044\u306a\u3044\u5834\u5408\u306f <em>Adam</em> \u306b\u30d5\u30a9\u30fc\u30eb\u30d0\u30c3\u30af\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> from <em>Adam</em> </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span><em>\u30a2\u30c0\u30e0\u304b\u3089\u5165\u624b\u3057\u3066</em></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span>.</p>\n<p>\ud83d\uddd2 The paper uses the notation <span translate=no>_^_1_^_</span> for this, which we don't use that here because it confuses with the Adam's usage of the same notation for bias corrected exponential moving average. </p>\n": "<p>\u53d6\u5f97<span translate=no>_^_0_^_</span>\u3002</p>\n<p>\ud83d\uddd2 <span translate=no>_^_1_^_</span> \u3053\u306e\u8ad6\u6587\u3067\u306f\u3053\u306e\u8868\u8a18\u6cd5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059\u304c\u3001\u3053\u3053\u3067\u306f\u4f7f\u7528\u3057\u307e\u305b\u3093\u3002\u3053\u308c\u306f\u3001\u30a2\u30c0\u30e0\u304c\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3055\u308c\u305f\u6307\u6570\u79fb\u52d5\u5e73\u5747\u306b\u3064\u3044\u3066\u540c\u3058\u8868\u8a18\u6cd5\u3092\u4f7f\u7528\u3059\u308b\u3053\u3068\u3068\u6df7\u540c\u3059\u308b\u305f\u3081\u3067\u3059\u3002</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> flag is <span translate=no>_^_1_^_</span> for this parameter group, we maintain the maximum of exponential moving average of squared gradient </p>\n": "<p><span translate=no>_^_0_^_</span>\u3053\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306b\u30d5\u30e9\u30b0\u3092\u6307\u5b9a\u3059\u308b\u3068\u3001\u4e8c\u4e57\u52fe\u914d\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u306e\u6700\u5927\u5024\u304c\u7dad\u6301\u3055\u308c\u307e\u3059\u3002<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>If this parameter group is using <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3053\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u304c\u4f7f\u7528\u3057\u3066\u3044\u308b\u5834\u5408 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Initialize the relevant optimizer </p>\n": "<p>\u95a2\u9023\u3059\u308b\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u521d\u671f\u5316\u3057\u307e\u3059</p>\n",
|
||||
"<p>Make sure <span translate=no>_^_0_^_</span> </p>\n": "<p>\u78ba\u8a8d\u3057\u3066 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Optimal, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u9069\u3001<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Optimize </p>\n": "<p>\u6700\u9069\u5316</p>\n",
|
||||
"<p>Run for <span translate=no>_^_0_^_</span> steps </p>\n": "<p><span translate=no>_^_0_^_</span>\u30e9\u30f3\u30cb\u30f3\u30b0\u30fb\u30d5\u30a9\u30fc\u30fb\u30b9\u30c6\u30c3\u30d7\u30b9</p>\n",
|
||||
"<p>Run the synthetic experiment is <em>AMSGrad</em> You can see that AMSGrad converges to true optimal <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5408\u6210\u5b9f\u9a13\u3092\u5b9f\u884c\u3059\u308b\u3068amsGrad\u3067\u3059\u3002<em>amsGrad\u304c\u771f\u6700\u9069\u306b\u53ce\u675f\u3059\u308b\u3053\u3068\u304c\u308f\u304b\u308a\u307e\u3059</em>\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Run the synthetic experiment is <em>Adam</em>. You can see that Adam converges at <span translate=no>_^_0_^_</span> </p>\n": "<p><em>\u5408\u6210\u5b9f\u9a13\u3092\u5b9f\u884c\u3059\u308b\u306e\u306f\u30a2\u30c0\u30e0\u3067\u3059</em>\u3002\u30a2\u30c0\u30e0\u304c\u6b21\u306e\u5834\u6240\u306b\u53ce\u675f\u3057\u3066\u3044\u308b\u306e\u304c\u308f\u304b\u308a\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Track results every 1,000 steps </p>\n": "<p>1,000 \u30b9\u30c6\u30c3\u30d7\u3054\u3068\u306b\u7d50\u679c\u3092\u30c8\u30e9\u30c3\u30ad\u30f3\u30b0</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of AMSGrad optimizer.": "AMSgrad \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u7c21\u5358\u306a PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"AMSGrad Optimizer": "\u30de\u30b9\u30b0\u30e9\u30fc\u30c9\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"<h1>AMSGrad</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/1904.09237\">On the Convergence of Adam and Beyond</a>.</p>\n<p>We implement this as an extension to our <a href=\"adam.html\">Adam optimizer implementation</a>. The implementation it self is really small since it's very similar to Adam.</p>\n<p>We also have an implementation of the synthetic example described in the paper where Adam fails to converge.</p>\n": "<h1>\u963f\u59c6\u65af\u683c\u62c9\u5fb7</h1>\n<p>\u8fd9\u662f <a href=\"https://pytorch.org\">PyTorch</a> \u5bf9\u300a<a href=\"https://arxiv.org/abs/1904.09237\">\u4e9a\u5f53\u4e0e\u8d85\u8d8a\u7684\u878d\u5408\u300b\u4e00\u6587\u7684</a>\u5b9e\u73b0\u3002</p>\n<p>\u6211\u4eec\u5c06\u5176\u4f5c\u4e3a\u6211\u4eec\u7684 <a href=\"adam.html\">Adam \u4f18\u5316\u5668\u5b9e\u73b0</a>\u7684\u6269\u5c55\u3002\u5b83\u81ea\u8eab\u7684\u5b9e\u73b0\u975e\u5e38\u5c0f\uff0c\u56e0\u4e3a\u5b83\u4e0e\u4e9a\u5f53\u975e\u5e38\u76f8\u4f3c\u3002</p>\n<p>\u6211\u4eec\u8fd8\u5b9e\u73b0\u4e86\u672c\u6587\u4e2d\u63cf\u8ff0\u7684\u5408\u6210\u793a\u4f8b\uff0c\u5176\u4e2d\u4e9a\u5f53\u672a\u80fd\u6536\u655b\u3002</p>\n",
|
||||
"<h2>AMSGrad Optimizer</h2>\n<p>This class extends from Adam optimizer defined in <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>. Adam optimizer is extending the class <span translate=no>_^_1_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_2_^_</span></a>.</p>\n": "<h2>amsGrad \u4f18\u5316\u5668</h2>\n<p>\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 Adam \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684<a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>\u3002Adam \u4f18\u5316\u5668\u6b63\u5728\u6269\u5c55\u4e2d<span translate=no>_^_1_^_</span>\u5b9a\u4e49\u7684\u7c7b<a href=\"index.html\"><span translate=no>_^_2_^_</span></a>\u3002</p>\n",
|
||||
"<h2>Synthetic Experiment</h2>\n<p>This is the synthetic experiment described in the paper, that shows a scenario where <em>Adam</em> fails.</p>\n<p>The paper (and Adam) formulates the problem of optimizing as minimizing the expected value of a function, <span translate=no>_^_0_^_</span> with respect to the parameters <span translate=no>_^_1_^_</span>. In the stochastic training setting we do not get hold of the function <span translate=no>_^_2_^_</span> it self; that is, when you are optimizing a NN <span translate=no>_^_3_^_</span> would be the function on entire batch of data. What we actually evaluate is a mini-batch so the actual function is realization of the stochastic <span translate=no>_^_4_^_</span>. This is why we are talking about an expected value. So let the function realizations be <span translate=no>_^_5_^_</span> for each time step of training.</p>\n<p>We measure the performance of the optimizer as the regret, <span translate=no>_^_6_^_</span> where <span translate=no>_^_7_^_</span> is the parameters at time step <span translate=no>_^_8_^_</span>, and <span translate=no>_^_9_^_</span> is the optimal parameters that minimize <span translate=no>_^_10_^_</span>.</p>\n<p>Now lets define the synthetic problem,</p>\n<span translate=no>_^_11_^_</span><p>where <span translate=no>_^_12_^_</span>. The optimal solution is <span translate=no>_^_13_^_</span>.</p>\n<p>This code will try running <em>Adam</em> and <em>AMSGrad</em> on this problem.</p>\n": "<h2>\u5408\u6210\u5b9e\u9a8c</h2>\n<p>\u8fd9\u662f\u8bba\u6587\u4e2d\u63cf\u8ff0\u7684\u5408\u6210\u5b9e\u9a8c\uff0c\u5b83\u663e\u793a\u4e86<em>\u4e9a\u5f53</em>\u5931\u8d25\u7684\u60c5\u666f\u3002</p>\n<p>\u672c\u6587\uff08\u548c\u4e9a\u5f53\uff09\u5c06\u4f18\u5316\u95ee\u9898\u63cf\u8ff0\u4e3a\u6700\u5c0f\u5316\u51fd\u6570<span translate=no>_^_0_^_</span>\u76f8\u5bf9\u4e8e\u53c2\u6570\u7684\u9884\u671f\u503c<span translate=no>_^_1_^_</span>\u3002\u5728\u968f\u673a\u8bad\u7ec3\u8bbe\u7f6e\u4e2d\uff0c\u6211\u4eec\u65e0\u6cd5\u638c\u63e1<span translate=no>_^_2_^_</span>\u5b83\u81ea\u8eab\u7684\u51fd\u6570\uff1b\u4e5f\u5c31\u662f\u8bf4\uff0c\u5f53\u4f60\u4f18\u5316\u65f6\uff0cNN<span translate=no>_^_3_^_</span> \u5c06\u662f\u6574\u6279\u6570\u636e\u7684\u51fd\u6570\u3002\u6211\u4eec\u5b9e\u9645\u8bc4\u4f30\u7684\u662f\u4e00\u4e2a\u5c0f\u6279\u91cf\uff0c\u6240\u4ee5\u5b9e\u9645\u7684\u529f\u80fd\u662f\u968f\u673a\u6307\u6807\u7684\u5b9e\u73b0<span translate=no>_^_4_^_</span>\u3002\u8fd9\u5c31\u662f\u6211\u4eec\u8c08\u8bba\u9884\u671f\u503c\u7684\u539f\u56e0\u3002\u56e0\u6b64\uff0c\u8ba9\u51fd\u6570\u5b9e\u73b0<span translate=no>_^_5_^_</span>\u9002\u7528\u4e8e\u8bad\u7ec3\u7684\u6bcf\u4e2a\u65f6\u95f4\u6b65\u3002</p>\n<p>\u6211\u4eec\u5c06\u4f18\u5316\u5668\u7684\u6027\u80fd\u4f5c\u4e3a\u9057\u61be\u6765\u8861\u91cf\uff0c<span translate=no>_^_6_^_</span>\u5176\u4e2d<span translate=no>_^_7_^_</span>\u662f\u65f6\u95f4\u6b65\u7684\u53c2\u6570<span translate=no>_^_8_^_</span>\uff0c<span translate=no>_^_9_^_</span>\u662f\u6700\u4f73\u7684\u6700\u5c0f\u5316\u7684\u53c2\u6570<span translate=no>_^_10_^_</span>\u3002</p>\n<p>\u73b0\u5728\u8ba9\u6211\u4eec\u6765\u5b9a\u4e49\u7efc\u5408\u95ee\u9898\uff0c</p>\n<span translate=no>_^_11_^_</span><p>\u5728\u54ea\u91cc<span translate=no>_^_12_^_</span>\u3002\u6700\u4f73\u7684\u89e3\u51b3\u65b9\u6848\u662f<span translate=no>_^_13_^_</span>\u3002</p>\n<p>\u8fd9\u6bb5\u4ee3\u7801\u5c06\u5c1d\u8bd5\u8fd0\u884c<em>\u4e9a\u5f53</em>\u548c<em>\u963f\u59c6\u65af\u683c</em>\u62c9\u5fb7\u6765\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\u3002</p>\n",
|
||||
"<h3><span translate=no>_^_0_^_</span></h3>\n": "<h3><span translate=no>_^_0_^_</span></h3>\n",
|
||||
"<h3>Calculate <span translate=no>_^_0_^_</span> and and <span translate=no>_^_1_^_</span> or <span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_4_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_5_^_</span> is the current gradient tensor <span translate=no>_^_6_^_</span> for the parameter <span translate=no>_^_7_^_</span></li></ul>\n": "<h3>\u8ba1\u7b97<span translate=no>_^_0_^_</span>\u548c\u548c<span translate=no>_^_1_^_</span>\u6216<span translate=no>_^_2_^_</span></h3>\n<ul><li><span translate=no>_^_3_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_4_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_5_^_</span>\u662f\u53c2\u6570\u7684\u5f53\u524d\u68af<span translate=no>_^_6_^_</span>\u5ea6\u5f20\u91cf<span translate=no>_^_7_^_</span></li></ul>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>\u521d\u59cb\u5316\u53c2\u6570\u72b6\u6001</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_16_^_</span>.</li></ul>\n": "<h3>\u521d\u59cb\u5316\u4f18\u5316\u5668</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\u5217\u8868</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5b66\u4e60\u7387<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f (<span translate=no>_^_4_^_</span>,<span translate=no>_^_5_^_</span>) \u7684\u5143\u7ec4</li>\n<li><span translate=no>_^_6_^_</span>\u662f<span translate=no>_^_7_^_</span>\u6216<span translate=no>_^_8_^_</span>\u57fa\u4e8e<span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span>\u662f\u5728\u4e2d<span translate=no>_^_11_^_</span>\u5b9a\u4e49\u7684\u7c7b\u7684\u5b9e\u4f8b <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li>\u201coptimized_update\u201d \u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u5728\u6dfb\u52a0\u540e\u662f\u5426\u8981\u4f18\u5316\u7b2c\u4e8c\u4e2a\u65f6\u523b\u7684\u504f\u5dee\u6821\u6b63<span translate=no>_^_13_^_</span></li>\n<li><span translate=no>_^_14_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u6307\u793a\u662f\u4f7f\u7528 AmsGrad \u8fd8\u662f\u56de\u9000\u5230\u666e\u901a\u7684 Adam</li>\n<li><span translate=no>_^_15_^_</span>\u662f\u7ec4\u503c\u7684\u9ed8\u8ba4\u5b57\u5178\u3002\u5f53\u4f60\u60f3\u6269\u5c55\u7c7b\u65f6\uff0c\u8fd9\u5f88\u6709\u7528<span translate=no>_^_16_^_</span>\u3002</li></ul>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span>.</p>\n<p>\ud83e\udd14 I feel you should be taking / maintaining the max of the bias corrected second exponential average of squared gradient. But this is how it's <a href=\"https://github.com/pytorch/pytorch/blob/19f4c5110e8bcad5e7e75375194262fca0a6293a/torch/optim/functional.py#L90\">implemented in PyTorch also</a>. I guess it doesn't really matter since bias correction only increases the value and it only makes an actual difference during the early few steps of the training. </p>\n": "<p>\u8ba1\u7b97<span translate=no>_^_0_^_</span>\u3002</p>\n<p>\ud83e\udd14 \u6211\u89c9\u5f97\u4f60\u5e94\u8be5\u53d6/\u4fdd\u6301\u504f\u5dee\u6821\u6b63\u7684\u5e73\u65b9\u68af\u5ea6\u7684\u7b2c\u4e8c\u4e2a\u6307\u6570\u5e73\u5747\u503c\u7684\u6700\u5927\u503c\u3002\u4f46\u8fd9\u4e5f\u662f<a href=\"https://github.com/pytorch/pytorch/blob/19f4c5110e8bcad5e7e75375194262fca0a6293a/torch/optim/functional.py#L90\">\u5728 PyTorch \u4e2d\u5b9e\u73b0</a>\u5b83\u7684\u65b9\u5f0f\u3002\u6211\u60f3\u8fd9\u5e76\u4e0d\u91cd\u8981\uff0c\u56e0\u4e3a\u504f\u5dee\u6821\u6b63\u53ea\u4f1a\u589e\u52a0\u503c\uff0c\u800c\u4e14\u53ea\u4f1a\u5728\u8bad\u7ec3\u7684\u6700\u521d\u51e0\u4e2a\u6b65\u9aa4\u4e2d\u4ea7\u751f\u5b9e\u9645\u5dee\u5f02\u3002</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"<p>Call <span translate=no>_^_0_^_</span> of Adam optimizer which we are extending </p>\n": "<p>\u6211\u4eec\u6b63\u5728\u6269\u5c55<span translate=no>_^_0_^_</span>\u7684 Call of Adam \u4f18\u5316\u5668</p>\n",
|
||||
"<p>Clear gradients </p>\n": "<p>\u6e10\u53d8\u6e05\u6670</p>\n",
|
||||
"<p>Create experiment to record results </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c\u4ee5\u8bb0\u5f55\u7ed3\u679c</p>\n",
|
||||
"<p>Define <span translate=no>_^_0_^_</span> parameter </p>\n": "<p>\u5b9a\u4e49<span translate=no>_^_0_^_</span>\u53c2\u6570</p>\n",
|
||||
"<p>Fall back to <em>Adam</em> if the parameter group is not using <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u53c2\u6570\u7ec4\u672a\u4f7f\u7528\uff0c\u5219\u56de\u9000\u5230 <em>Adam</em><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> from <em>Adam</em> </p>\n": "<p><span translate=no>_^_1_^_</span>\u4ece <em>Adam</em> \u90a3\u91cc\u5f97<span translate=no>_^_0_^_</span>\u5230</p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span>.</p>\n<p>\ud83d\uddd2 The paper uses the notation <span translate=no>_^_1_^_</span> for this, which we don't use that here because it confuses with the Adam's usage of the same notation for bias corrected exponential moving average. </p>\n": "<p>\u5f97\u5230<span translate=no>_^_0_^_</span>\u3002</p>\n<p>\ud83d\uddd2 \u672c\u6587\u4f7f\u7528\u4e86\u8fd9\u4e2a\u7b26\u53f7<span translate=no>_^_1_^_</span>\uff0c\u6211\u4eec\u5728\u8fd9\u91cc\u4e0d\u4f7f\u7528\u8fd9\u79cd\u7b26\u53f7\uff0c\u56e0\u4e3a\u5b83\u4e0e\u4e9a\u5f53\u5bf9\u504f\u5dee\u6821\u6b63\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u4f7f\u7528\u76f8\u540c\u7684\u7b26\u53f7\u6df7\u6dc6\u4e86\u3002</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> flag is <span translate=no>_^_1_^_</span> for this parameter group, we maintain the maximum of exponential moving average of squared gradient </p>\n": "<p>\u5982\u679c f<span translate=no>_^_0_^_</span> lag<span translate=no>_^_1_^_</span> \u7528\u4e8e\u6b64\u53c2\u6570\u7ec4\uff0c\u5219\u6211\u4eec\u4fdd\u6301\u68af\u5ea6\u5e73\u65b9\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u7684\u6700\u5927\u503c</p>\n",
|
||||
"<p>If this parameter group is using <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u6b64\u53c2\u6570\u7ec4\u6b63\u5728\u4f7f\u7528<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Initialize the relevant optimizer </p>\n": "<p>\u521d\u59cb\u5316\u76f8\u5173\u7684\u4f18\u5316\u5668</p>\n",
|
||||
"<p>Make sure <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8bf7\u786e\u4fdd<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Optimal, <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u4f73\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Optimize </p>\n": "<p>\u4f18\u5316</p>\n",
|
||||
"<p>Run for <span translate=no>_^_0_^_</span> steps </p>\n": "<p>\u8dd1\u6b65\u8dd1<span translate=no>_^_0_^_</span>\u6b65</p>\n",
|
||||
"<p>Run the synthetic experiment is <em>AMSGrad</em> You can see that AMSGrad converges to true optimal <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5728 <em>amsGrad</em> \u8fd0\u884c\u5408\u6210\u5b9e\u9a8c\u4f60\u53ef\u4ee5\u770b\u5230 amsGrad \u4f1a\u805a\u5230\u771f\u6b63\u7684\u6700\u4f18\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Run the synthetic experiment is <em>Adam</em>. You can see that Adam converges at <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8fd0\u884c\u5408\u6210\u5b9e\u9a8c\u7684\u662f<em>\u4e9a\u5f53</em>\u3002\u4f60\u53ef\u4ee5\u770b\u5230\u4e9a\u5f53\u805a\u96c6\u5728<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Track results every 1,000 steps </p>\n": "<p>\u6bcf 1000 \u6b65\u8ddf\u8e2a\u4e00\u6b21\u7ed3\u679c</p>\n",
|
||||
"A simple PyTorch implementation/tutorial of AMSGrad optimizer.": "\u4e00\u4e2a\u7b80\u5355\u7684 AmsGrad \u4f18\u5316\u5668\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002",
|
||||
"AMSGrad Optimizer": "amsGrad \u4f18\u5316\u5668"
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"<h1>Configurable Optimizer</h1>\n": "<h1>\u8a2d\u5b9a\u53ef\u80fd\u306a\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h1>\n",
|
||||
"<p> <a id=\"OptimizerConfigs\"></a></p>\n<h2>Optimizer Configurations</h2>\n": "<p><a id=\"OptimizerConfigs\"></a></p>\n<h2>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u69cb\u6210</h2>\n",
|
||||
"<p>Beta values <span translate=no>_^_0_^_</span> for Adam </p>\n": "<p><span translate=no>_^_0_^_</span>\u30a2\u30c0\u30e0\u306e\u30d9\u30fc\u30bf\u5024</p>\n",
|
||||
"<p>Epsilon <span translate=no>_^_0_^_</span> for adam </p>\n": "<p>\u30a4\u30d7\u30b7\u30ed\u30f3\u30fb\u30d5\u30a9\u30fc\u30fb\u30a2\u30c0\u30e0 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5b66\u7fd2\u7387 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Model embedding size for Noam optimizer </p>\n": "<p>Noam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u30e2\u30c7\u30eb\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Momentum for SGD </p>\n": "<p>\u30b7\u30f3\u30ac\u30dd\u30fc\u30eb\u30c9\u30eb\u306e\u30e2\u30e1\u30f3\u30bf\u30e0</p>\n",
|
||||
"<p>Number of warmup optimizer steps </p>\n": "<p>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u30b9\u30c6\u30c3\u30d7\u6570</p>\n",
|
||||
"<p>Optimizer </p>\n": "<p>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</p>\n",
|
||||
"<p>Parameters to be optimized </p>\n": "<p>\u6700\u9069\u5316\u3059\u308b\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc</p>\n",
|
||||
"<p>Total number of optimizer steps (for cosine decay) </p>\n": "<p>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30b9\u30c6\u30c3\u30d7\u306e\u7dcf\u6570 (\u30b3\u30b5\u30a4\u30f3\u6e1b\u8870\u7528)</p>\n",
|
||||
"<p>Weight decay </p>\n": "<p>\u4f53\u91cd\u6e1b\u5c11</p>\n",
|
||||
"<p>Whether the adam update is optimized (different epsilon) </p>\n": "<p>adam\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u304c\u6700\u9069\u5316\u3055\u308c\u3066\u3044\u308b\u304b\u3069\u3046\u304b (\u30a4\u30d7\u30b7\u30ed\u30f3\u304c\u7570\u306a\u308b)</p>\n",
|
||||
"<p>Whether to degenerate to SGD in AdaBelief </p>\n": "<p>AdabElief \u3067 SGD \u306b\u7e2e\u9000\u3055\u305b\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>Whether to use AMSGrad </p>\n": "<p>\u30a2\u30e0\u30b9\u30b0\u30e9\u30fc\u30c9\u3092\u4f7f\u7528\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>Whether to use Rectified Adam in AdaBelief </p>\n": "<p>AdabElief \u3067\u30ec\u30af\u30c6\u30a3\u30d5\u30a1\u30a4\u30c9\u30a2\u30c0\u30e0\u3092\u4f7f\u7528\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>Whether weight decay is absolute or should be multiplied by learning rate </p>\n": "<p>\u6e1b\u91cf\u304c\u7d76\u5bfe\u4f53\u91cd\u306a\u306e\u304b\u3001\u305d\u308c\u3068\u3082\u5b66\u7fd2\u7387\u3067\u639b\u3051\u308b\u3079\u304d\u306a\u306e\u304b</p>\n",
|
||||
"<p>Whether weight decay is decoupled; i.e. weight decay is not added to gradients </p>\n": "<p>\u91cd\u307f\u6e1b\u8870\u304c\u5207\u308a\u96e2\u3055\u308c\u3066\u3044\u308b\u304b\u3069\u3046\u304b\u3001\u3064\u307e\u308a\u91cd\u307f\u6e1b\u8870\u304c\u52fe\u914d\u306b\u52a0\u3048\u3089\u308c\u306a\u3044\u304b\u3069\u3046\u304b</p>\n",
|
||||
"Configurable optimizer module": "\u8a2d\u5b9a\u53ef\u80fd\u306a\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30e2\u30b8\u30e5\u30fc\u30eb",
|
||||
"This implements a configurable module for optimizers.": "\u3053\u308c\u306b\u3088\u308a\u3001\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u7528\u306e\u8a2d\u5b9a\u53ef\u80fd\u306a\u30e2\u30b8\u30e5\u30fc\u30eb\u304c\u5b9f\u88c5\u3055\u308c\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"<h1>Configurable Optimizer</h1>\n": "<h1>\u0db8\u0dcf\u0db1\u0d9a\u0dbd\u0dc4\u0dd0\u0d9a\u0dd2 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h1>\n",
|
||||
"<p> <a id=\"OptimizerConfigs\"></a></p>\n<h2>Optimizer Configurations</h2>\n": "<p> <a id=\"OptimizerConfigs\"></a></p>\n<h2>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h2>\n",
|
||||
"<p>Beta values <span translate=no>_^_0_^_</span> for Adam </p>\n": "<p>\u0d86\u0daf\u0db8\u0dca <span translate=no>_^_0_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db6\u0dd3\u0da7\u0dcf \u0d85\u0d9c\u0dba\u0db1\u0dca </p>\n",
|
||||
"<p>Epsilon <span translate=no>_^_0_^_</span> for adam </p>\n": "<p>\u0d86\u0daf\u0db8\u0dca <span translate=no>_^_0_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0db4\u0dca\u0dc3\u0dd2\u0dbd\u0db1\u0dca </p>\n",
|
||||
"<p>Learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Model embedding size for Noam optimizer </p>\n": "<p>Noam\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Momentum for SGD </p>\n": "<p>SGD\u0dc3\u0db3\u0dc4\u0dcf \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Number of warmup optimizer steps </p>\n": "<p>\u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Optimizer </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba </p>\n",
|
||||
"<p>Parameters to be optimized </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca </p>\n",
|
||||
"<p>Total number of optimizer steps (for cosine decay) </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 (\u0d9a\u0ddc\u0dc3\u0dba\u0dd2\u0db1\u0dca \u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf) </p>\n",
|
||||
"<p>Weight decay </p>\n": "<p>\u0dc3\u0dd2\u0dbb\u0dd4\u0dbb\u0dda\u0db6\u0dbb \u0d9a\u0dca\u0dc2\u0dba </p>\n",
|
||||
"<p>Whether the adam update is optimized (different epsilon) </p>\n": "<p>\u0d86\u0daf\u0db8\u0dca\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc0\u0dd3\u0db8\u0dad \u0dba\u0db1\u0dca\u0db1 (\u0dc0\u0dd2\u0dc0\u0dd2\u0db0 epsilon) </p>\n",
|
||||
"<p>Whether to degenerate to SGD in AdaBelief </p>\n": "<p>Adeabelief\u0dc4\u0dd2 SGD \u0dc0\u0dd9\u0dad \u0db4\u0dbb\u0dd2\u0dc4\u0dcf\u0db1\u0dd2\u0dba\u0da7 \u0db4\u0dad\u0dca \u0dc0\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether to use AMSGrad </p>\n": "<p>AMSGrad\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether to use Rectified Adam in AdaBelief </p>\n": "<p>Adamelief\u0dc4\u0dd2 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d86\u0daf\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether weight decay is absolute or should be multiplied by learning rate </p>\n": "<p>\u0db6\u0dbb\u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0db1\u0dd2\u0dbb\u0db4\u0dda\u0d9a\u0dca\u0dc2 \u0dc4\u0ddd \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0d9c\u0dd4\u0dab \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether weight decay is decoupled; i.e. weight decay is not added to gradients </p>\n": "<p>\u0db6\u0dbb\u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0daf\u0dd2\u0dbb\u0dcf\u0db4\u0dad\u0dca \u0dc0\u0dda\u0daf; \u0d91\u0db1\u0db8\u0dca \u0db6\u0dbb \u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc0\u0dbd\u0da7 \u0d91\u0d9a\u0dad\u0dd4 \u0db1\u0ddc\u0dc0\u0dda </p>\n",
|
||||
"Configurable optimizer module": "\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba",
|
||||
"This implements a configurable module for optimizers.": "\u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba\u0d9a\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"<h1>Configurable Optimizer</h1>\n": "<h1>\u53ef\u914d\u7f6e\u7684\u4f18\u5316\u5668</h1>\n",
|
||||
"<p> <a id=\"OptimizerConfigs\"></a></p>\n<h2>Optimizer Configurations</h2>\n": "<p><a id=\"OptimizerConfigs\"></a></p>\n<h2>\u4f18\u5316\u5668\u914d\u7f6e</h2>\n",
|
||||
"<p>Beta values <span translate=no>_^_0_^_</span> for Adam </p>\n": "<p>Adam<span translate=no>_^_0_^_</span> \u7684 Beta \u503c</p>\n",
|
||||
"<p>Epsilon <span translate=no>_^_0_^_</span> for adam </p>\n": "<p>Epsilon<span translate=no>_^_0_^_</span> \u4ee3\u8868\u4e9a\u5f53</p>\n",
|
||||
"<p>Learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5b66\u4e60\u7387<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Model embedding size for Noam optimizer </p>\n": "<p>Noam \u4f18\u5316\u5668\u7684\u6a21\u578b\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Momentum for SGD </p>\n": "<p>\u65b0\u52a0\u5761\u5143\u7684\u52bf\u5934</p>\n",
|
||||
"<p>Number of warmup optimizer steps </p>\n": "<p>\u9884\u70ed\u4f18\u5316\u5668\u6b65\u9aa4\u6570</p>\n",
|
||||
"<p>Optimizer </p>\n": "<p>\u4f18\u5316\u5668</p>\n",
|
||||
"<p>Parameters to be optimized </p>\n": "<p>\u8981\u4f18\u5316\u7684\u53c2\u6570</p>\n",
|
||||
"<p>Total number of optimizer steps (for cosine decay) </p>\n": "<p>\u4f18\u5316\u5668\u6b65\u957f\u603b\u6570\uff08\u4f59\u5f26\u8870\u51cf\uff09</p>\n",
|
||||
"<p>Weight decay </p>\n": "<p>\u4f53\u91cd\u8870\u51cf</p>\n",
|
||||
"<p>Whether the adam update is optimized (different epsilon) </p>\n": "<p>adam \u66f4\u65b0\u662f\u5426\u7ecf\u8fc7\u4f18\u5316\uff08\u4e0d\u540c\u7684 epsilon\uff09</p>\n",
|
||||
"<p>Whether to degenerate to SGD in AdaBelief </p>\n": "<p>\u662f\u5426\u5728 AdaBeLief \u4e2d\u9000\u5316\u4e3a\u65b0\u52a0\u5761\u5143</p>\n",
|
||||
"<p>Whether to use AMSGrad </p>\n": "<p>\u662f\u5426\u4f7f\u7528 AmsGrad</p>\n",
|
||||
"<p>Whether to use Rectified Adam in AdaBelief </p>\n": "<p>\u662f\u5426\u5728 AdaBelief \u4e2d\u4f7f\u7528\u6574\u6539\u8fc7\u7684\u4e9a\u5f53</p>\n",
|
||||
"<p>Whether weight decay is absolute or should be multiplied by learning rate </p>\n": "<p>\u4f53\u91cd\u8870\u51cf\u662f\u7edd\u5bf9\u7684\u8fd8\u662f\u5e94\u8be5\u4e58\u4ee5\u5b66\u4e60\u901f\u7387</p>\n",
|
||||
"<p>Whether weight decay is decoupled; i.e. weight decay is not added to gradients </p>\n": "<p>\u6743\u91cd\u8870\u51cf\u662f\u5426\u89e3\u8026\uff1b\u5373\u6743\u91cd\u8870\u51cf\u4e0d\u6dfb\u52a0\u5230\u68af\u5ea6\u4e2d</p>\n",
|
||||
"Configurable optimizer module": "\u53ef\u914d\u7f6e\u7684\u4f18\u5316\u5668\u6a21\u5757",
|
||||
"This implements a configurable module for optimizers.": "\u8fd9\u4e3a\u4f18\u5316\u5668\u5b9e\u73b0\u4e86\u4e00\u4e2a\u53ef\u914d\u7f6e\u7684\u6a21\u5757\u3002"
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>MNIST example to test the optimizers</h1>\n": "<h1>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u30c6\u30b9\u30c8\u3059\u308b\u305f\u3081\u306e MNIST \u306e\u4f8b</h1>\n",
|
||||
"<h2>Configurable Experiment Definition</h2>\n": "<h2>\u8a2d\u5b9a\u53ef\u80fd\u306a\u5b9f\u9a13\u5b9a\u7fa9</h2>\n",
|
||||
"<h2>The model</h2>\n": "<h2>\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<p> Create a configurable optimizer. We can change the optimizer type and hyper-parameters using configurations.</p>\n": "<p>\u8a2d\u5b9a\u53ef\u80fd\u306a\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f5c\u6210\u3057\u307e\u3059\u3002\u69cb\u6210\u3092\u4f7f\u7528\u3057\u3066\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u306e\u30bf\u30a4\u30d7\u3068\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u5909\u66f4\u3067\u304d\u307e\u3059</p>\u3002\n",
|
||||
"<p>Add global step if we are in training mode </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30e2\u30fc\u30c9\u306e\u5834\u5408\u306f\u30b0\u30ed\u30fc\u30d0\u30eb\u30b9\u30c6\u30c3\u30d7\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Calculate the accuracy </p>\n": "<p>\u7cbe\u5ea6\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate the gradients </p>\n": "<p>\u52fe\u914d\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate the loss </p>\n": "<p>\u640d\u5931\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30af\u30ea\u30a2</p>\n",
|
||||
"<p>Get the batch </p>\n": "<p>\u30d0\u30c3\u30c1\u3092\u5165\u624b</p>\n",
|
||||
"<p>Log the loss </p>\n": "<p>\u640d\u5931\u3092\u8a18\u9332\u3059\u308b</p>\n",
|
||||
"<p>Log the parameter and gradient L2 norms once per epoch </p>\n": "<p>\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3068\u52fe\u914d\u306e L2 \u30ce\u30eb\u30e0\u3092\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b 1 \u56de\u8a18\u9332\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Optimize if we are in training mode </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30e2\u30fc\u30c9\u306e\u5834\u5408\u306f\u6700\u9069\u5316</p>\n",
|
||||
"<p>Run the model and specify whether to log the activations </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u5b9f\u884c\u3057\u3001\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u3092\u30ed\u30b0\u306b\u8a18\u9332\u3059\u308b\u304b\u3069\u3046\u304b\u3092\u6307\u5b9a\u3057\u307e\u3059</p>\n",
|
||||
"<p>Save logs </p>\n": "<p>\u30ed\u30b0\u3092\u4fdd\u5b58</p>\n",
|
||||
"<p>Specify the optimizer </p>\n": "<p>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u6307\u5b9a</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u6700\u9069\u5316\u306e\u4e00\u6b69\u3092\u8e0f\u307f\u51fa\u3059</p>\n",
|
||||
"MNIST example to test the optimizers": "\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u30c6\u30b9\u30c8\u3059\u308b\u305f\u3081\u306e MNIST \u306e\u4f8b",
|
||||
"This is a simple MNIST example with a CNN model to test the optimizers.": "\u3053\u308c\u306f\u3001CNN \u30e2\u30c7\u30eb\u3092\u4f7f\u7528\u3057\u3066\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u30c6\u30b9\u30c8\u3059\u308b\u7c21\u5358\u306a MNIST \u306e\u4f8b\u3067\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>MNIST example to test the optimizers</h1>\n": "<h1>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf MNIST \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba</h1>\n",
|
||||
"<h2>Configurable Experiment Definition</h2>\n": "<h2>\u0dc3\u0dd0\u0d9a\u0dc3\u0dd6\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0dda \u0d85\u0dbb\u0dca\u0dae \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd3\u0db8</h2>\n",
|
||||
"<h2>The model</h2>\n": "<h2>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<p> Create a configurable optimizer. We can change the optimizer type and hyper-parameters using configurations.</p>\n": "<p> \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad\u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1. \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd2\u0db1\u0dca \u0d85\u0db4\u0da7 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0dc0\u0dbb\u0dca\u0d9c\u0dba \u0dc3\u0dc4 \u0d85\u0db0\u0dd2 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. </p>\n",
|
||||
"<p>Add global step if we are in training mode </p>\n": "<p>\u0d85\u0db4\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db8\u0dcf\u0daf\u0dd2\u0dbd\u0dd2\u0dba\u0dda \u0dc3\u0dd2\u0da7\u0dd3 \u0db1\u0db8\u0dca \u0d9c\u0ddd\u0dbd\u0dd3\u0dba \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate the accuracy </p>\n": "<p>\u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate the gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9c\u0dab\u0db1\u0dba </p>\n",
|
||||
"<p>Calculate the loss </p>\n": "<p>\u0d85\u0dbd\u0dcf\u0db7\u0dba\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d89\u0dc0\u0dad\u0dca </p>\n",
|
||||
"<p>Get the batch </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the loss </p>\n": "<p>\u0d85\u0dbd\u0dcf\u0db7\u0dba\u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the parameter and gradient L2 norms once per epoch </p>\n": "<p>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba\u0dc3\u0dc4 \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda L2 \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba\u0db1\u0dca \u0d91\u0d9a\u0dca \u0dc0\u0dbb\u0d9a\u0dca \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Optimize if we are in training mode </p>\n": "<p>\u0d85\u0db4\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db8\u0dcf\u0daf\u0dd2\u0dbd\u0dd2\u0dba\u0dda \u0dc3\u0dd2\u0da7\u0dd3 \u0db1\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run the model and specify whether to log the activations </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 \u0dc3\u0db3\u0dc4\u0db1\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Save logs </p>\n": "<p>\u0dbd\u0ddc\u0d9c\u0dca\u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Specify the optimizer </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0db1\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"MNIST example to test the optimizers": "\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf MNIST \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba",
|
||||
"This is a simple MNIST example with a CNN model to test the optimizers.": "\u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dcf\u0dbb\u0d9a \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dd3\u0d91\u0db1\u0dca\u0d91\u0db1\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0dc4\u0dd2\u0dad \u0dc3\u0dbb\u0dbd MNIST \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>MNIST example to test the optimizers</h1>\n": "<h1>\u6d4b\u8bd5\u4f18\u5316\u5668\u7684 MNIST \u793a\u4f8b</h1>\n",
|
||||
"<h2>Configurable Experiment Definition</h2>\n": "<h2>\u53ef\u914d\u7f6e\u7684\u5b9e\u9a8c\u5b9a\u4e49</h2>\n",
|
||||
"<h2>The model</h2>\n": "<h2>\u8be5\u6a21\u578b</h2>\n",
|
||||
"<p> Create a configurable optimizer. We can change the optimizer type and hyper-parameters using configurations.</p>\n": "<p>\u521b\u5efa\u53ef\u914d\u7f6e\u7684\u4f18\u5316\u5668\u3002\u6211\u4eec\u53ef\u4ee5\u4f7f\u7528\u914d\u7f6e\u66f4\u6539\u4f18\u5316\u5668\u7c7b\u578b\u548c\u8d85\u53c2\u6570\u3002</p>\n",
|
||||
"<p>Add global step if we are in training mode </p>\n": "<p>\u5982\u679c\u6211\u4eec\u5904\u4e8e\u8bad\u7ec3\u6a21\u5f0f\uff0c\u5219\u6dfb\u52a0\u5168\u5c40\u6b65\u957f</p>\n",
|
||||
"<p>Calculate the accuracy </p>\n": "<p>\u8ba1\u7b97\u7cbe\u5ea6</p>\n",
|
||||
"<p>Calculate the gradients </p>\n": "<p>\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"<p>Calculate the loss </p>\n": "<p>\u8ba1\u7b97\u635f\u5931</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u6e05\u9664\u6e10\u53d8</p>\n",
|
||||
"<p>Get the batch </p>\n": "<p>\u83b7\u53d6\u6279\u6b21</p>\n",
|
||||
"<p>Log the loss </p>\n": "<p>\u8bb0\u5f55\u635f\u5931</p>\n",
|
||||
"<p>Log the parameter and gradient L2 norms once per epoch </p>\n": "<p>\u6bcf\u4e2a\u7eaa\u5143\u8bb0\u5f55\u4e00\u6b21\u53c2\u6570\u548c\u68af\u5ea6 L2 \u89c4\u8303</p>\n",
|
||||
"<p>Optimize if we are in training mode </p>\n": "<p>\u5982\u679c\u6211\u4eec\u5904\u4e8e\u8bad\u7ec3\u6a21\u5f0f\uff0c\u8bf7\u8fdb\u884c\u4f18\u5316</p>\n",
|
||||
"<p>Run the model and specify whether to log the activations </p>\n": "<p>\u8fd0\u884c\u6a21\u578b\u5e76\u6307\u5b9a\u662f\u5426\u8bb0\u5f55\u6fc0\u6d3b</p>\n",
|
||||
"<p>Save logs </p>\n": "<p>\u4fdd\u5b58\u65e5\u5fd7</p>\n",
|
||||
"<p>Specify the optimizer </p>\n": "<p>\u6307\u5b9a\u4f18\u5316\u5668</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u91c7\u53d6\u4f18\u5316\u5668\u6b65\u9aa4</p>\n",
|
||||
"MNIST example to test the optimizers": "\u6d4b\u8bd5\u4f18\u5316\u5668\u7684 MNIST \u793a\u4f8b",
|
||||
"This is a simple MNIST example with a CNN model to test the optimizers.": "\u8fd9\u662f\u4e00\u4e2a\u7b80\u5355\u7684 MNIST \u793a\u4f8b\uff0c\u5176\u4e2d\u5305\u542b CNN \u6a21\u578b\u6765\u6d4b\u8bd5\u4f18\u5316\u5668\u3002"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"<h1>Noam Optimizer</h1>\n<p>This is the <a href=\"https://pytorch.org\">PyTorch</a> implementation of optimizer introduced in the paper <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a>.</p>\n": "<h1>\u30ce\u30fc\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/1706.03762\">\u5fc5\u8981\u306a\u306e\u306f\u6ce8\u610f\u3060\u3051\u300d<a href=\"https://pytorch.org\">\u3067\u7d39\u4ecb\u3055\u308c\u3066\u3044\u308b\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306ePyTorch\u5b9f\u88c5\u3067\u3059</a></a>\u3002</p>\n",
|
||||
"<h2>Noam Optimizer</h2>\n<p>This class extends from Adam optimizer defined in <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>\u30ce\u30fc\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h2>\n<p>\u3053\u306e\u30af\u30e9\u30b9\u306f\u3001\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u62e1\u5f35\u3057\u305f\u3082\u306e\u3067\u3059\u3002<a href=\"adam.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u5b66\u7fd2\u7387\u3092\u53d6\u5f97</h3>\n<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u306f\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30b9\u30c6\u30c3\u30d7\u306e\u6570\u3067\u3059\u3002</p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> model size; i.e. number of dimensions in the transformer </li>\n<li><span translate=no>_^_17_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_18_^_</span>.</li></ul>\n": "<h3>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u30ea\u30b9\u30c8\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u5b66\u7fd2\u7387 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>(,) <span translate=no>_^_4_^_</span> \u306e\u30bf\u30d7\u30eb\u3067\u3059 <span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span>\u307e\u305f\u306f\u305d\u308c\u306b\u57fa\u3065\u3044\u3066\u3044\u308b <span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span><span translate=no>_^_11_^_</span>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3067\u3059 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li>'optimized_update'\u306f\u8ffd\u52a0\u5f8c\u306b\u884c\u3046\u3053\u3068\u3067\u30bb\u30ab\u30f3\u30c9\u30e2\u30fc\u30e1\u30f3\u30c8\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3092\u6700\u9069\u5316\u3059\u308b\u304b\u3069\u3046\u304b\u306e\u30d5\u30e9\u30b0\u3067\u3059 <span translate=no>_^_13_^_</span></li>\n<li><span translate=no>_^_14_^_</span>amsGrad\u3092\u4f7f\u7528\u3059\u308b\u304b\u3001\u30d7\u30ec\u30fc\u30f3\u306aAdam\u306b\u30d5\u30a9\u30fc\u30eb\u30d0\u30c3\u30af\u3059\u308b\u304b\u3092\u793a\u3059\u30d5\u30e9\u30b0\u3067\u3059</li>\n<li><span translate=no>_^_15_^_</span>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u30b9\u30c6\u30c3\u30d7\u6570</li>\n<li><span translate=no>_^_16_^_</span>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba\u3001\u3064\u307e\u308a\u5909\u5727\u5668\u306e\u6b21\u5143\u6570</li>\n<li><span translate=no>_^_17_^_</span>\u30b0\u30eb\u30fc\u30d7\u5024\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u8f9e\u66f8\u3067\u3059\u3002\u3053\u308c\u306f\u3001\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3059\u308b\u5834\u5408\u306b\u4fbf\u5229\u3067\u3059<span translate=no>_^_18_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Plot learning rate for different warmups and model sizes</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>\u3055\u307e\u3056\u307e\u306a\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u3068\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba\u306e\u5b66\u7fd2\u7387\u3092\u30d7\u30ed\u30c3\u30c8</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"Noam optimizer from Attention is All You Need paper": "\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30a4\u30ba\u30fb\u30aa\u30fc\u30eb\u30fb\u30e6\u30fc\u30fb\u30cb\u30fc\u30c9\u8ad6\u6587\u306e Noam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc",
|
||||
"This is a tutorial/implementation of Noam optimizer. Noam optimizer has a warm-up period and then an exponentially decaying learning rate.": "\u3053\u308c\u306fNoam\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb/\u5b9f\u88c5\u3067\u3059\u3002Noam Optimizer\u306b\u306f\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u671f\u9593\u304c\u3042\u308a\u3001\u305d\u306e\u5f8c\u306f\u5b66\u7fd2\u7387\u304c\u6307\u6570\u95a2\u6570\u7684\u306b\u4f4e\u4e0b\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"<h1>Noam Optimizer</h1>\n<p>This is the <a href=\"https://pytorch.org\">PyTorch</a> implementation of optimizer introduced in the paper <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a>.</p>\n": "<h1>\u0db1\u0dc0\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h1>\n<p>\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2\u0dad\u0dd4\u0dc5 \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd3 \u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dda <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/1706.03762\">\u0db8\u0dd9\u0dba\u0dba\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0d94\u0db6\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc3\u0dd2\u0dba\u0dbd\u0dca\u0dbd</a> . </p>\n",
|
||||
"<h2>Noam Optimizer</h2>\n<p>This class extends from Adam optimizer defined in <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>\u0db1\u0dc0\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0db8\u0dd9\u0db8\u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dd9\u0db1\u0dca \u0dc0\u0dd2\u0dc4\u0dd2\u0daf\u0dda <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>. </p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u0d89\u0d9c\u0dd9\u0db1\u0dd3\u0db8-\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1</h3>\n<p><span translate=no>_^_0_^_</span> \u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> model size; i.e. number of dimensions in the transformer </li>\n<li><span translate=no>_^_17_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_18_^_</span>.</li></ul>\n": "<h3>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0dba\u0dd2 </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dba\u0dd2 <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) \u0d9a tuple \u0dc0\u0dda </li>\n<li><span translate=no>_^_6_^_</span> <span translate=no>_^_7_^_</span> \u0dc4\u0ddd \u0db8\u0dad <span translate=no>_^_8_^_</span> \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> <span translate=no>_^_11_^_</span> \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba\u0dda \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0d9a\u0dd2 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update'\u0dba\u0db1\u0dd4 \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d91\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0daf\u0dd9\u0dc0\u0db1 \u0db8\u0ddc\u0dc4\u0ddc\u0dad\u0dda \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf \u0dba\u0db1\u0dca\u0db1 \u0db0\u0da2\u0dba\u0d9a\u0dd2 <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> \u0d86\u0daf\u0db8\u0dca \u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf AMSGrad \u0dc4\u0ddd \u0dc0\u0dd0\u0da7\u0dd3\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd9\u0db1 \u0db0\u0da2\u0dba\u0d9a\u0dd2 </li>\n<li><span translate=no>_^_15_^_</span> \u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_16_^_</span> \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba; i.e. \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0dda \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_17_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc1\u0db6\u0dca\u0daf \u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dd2. \u0d94\u0db6\u0da7 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dd2\u0da7 \u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dba\u0ddd\u0da2\u0db1\u0dc0\u0dad\u0dca <span translate=no>_^_18_^_</span>\u0dc0\u0dda. </li></ul>\n",
|
||||
"<h3>Plot learning rate for different warmups and model sizes</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>\u0dc0\u0dd2\u0dc0\u0dd2\u0db0\u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0dc3\u0dc4 \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dca\u0dbd\u0ddc\u0da7\u0dca \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"Noam optimizer from Attention is All You Need paper": "\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dd2\u0da7 Noam \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0d94\u0db6 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc3\u0dd2\u0dba\u0dbd\u0dd4 \u0dc0\u0dda",
|
||||
"This is a tutorial/implementation of Noam optimizer. Noam optimizer has a warm-up period and then an exponentially decaying learning rate.": "\u0db8\u0dd9\u0dba Noam \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0dda \u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba\u0d9a\u0dca/\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2. Noam \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0da7 \u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dcf\u0dbd \u0db4\u0dbb\u0dd2\u0da0\u0dca\u0da1\u0dda\u0daf\u0dba\u0d9a\u0dca \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0db4\u0dc3\u0dd4\u0dc0 on \u0dcf\u0dad\u0dd3\u0dba \u0dbd\u0dd9\u0dc3 \u0daf\u0dd2\u0dbb\u0dcf\u0db4\u0dad\u0dca \u0dc0\u0db1 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0d9a\u0dca \u0d87\u0dad."
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"<h1>Noam Optimizer</h1>\n<p>This is the <a href=\"https://pytorch.org\">PyTorch</a> implementation of optimizer introduced in the paper <a href=\"https://arxiv.org/abs/1706.03762\">Attention Is All You Need</a>.</p>\n": "<h1>Noam \u4f18\u5316\u5668</h1>\n<p>\u8fd9\u662f\u300a<a href=\"https://arxiv.org/abs/1706.03762\">\u6ce8\u610f\u5c31\u662f\u4f60\u6240\u9700\u8981\u7684\u300b\u4e00\u6587\u4e2d\u4ecb\u7ecd\u7684\u4f18\u5316\u5668\u7684</a> <a href=\"https://pytorch.org\">PyTorch</a> \u5b9e\u73b0\u3002</p>\n",
|
||||
"<h2>Noam Optimizer</h2>\n<p>This class extends from Adam optimizer defined in <a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>Noam \u4f18\u5316\u5668</h2>\n<p>\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 Adam \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684<a href=\"adam.html\"><span translate=no>_^_0_^_</span></a>\u3002</p>\n",
|
||||
"<h3>Get learning-rate</h3>\n<p><span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the number of warmup steps.</p>\n": "<h3>\u83b7\u53d6\u5b66\u4e60\u7387</h3>\n<p><span translate=no>_^_0_^_</span>\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u9884\u70ed\u6b65\u9aa4\u7684\u6570\u91cf\u3002</p>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li>'optimized_update' is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_13_^_</span> </li>\n<li><span translate=no>_^_14_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_15_^_</span> number of warmup steps </li>\n<li><span translate=no>_^_16_^_</span> model size; i.e. number of dimensions in the transformer </li>\n<li><span translate=no>_^_17_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_18_^_</span>.</li></ul>\n": "<h3>\u521d\u59cb\u5316\u4f18\u5316\u5668</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\u5217\u8868</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5b66\u4e60\u7387<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f (<span translate=no>_^_4_^_</span>,<span translate=no>_^_5_^_</span>) \u7684\u5143\u7ec4</li>\n<li><span translate=no>_^_6_^_</span>\u662f<span translate=no>_^_7_^_</span>\u6216<span translate=no>_^_8_^_</span>\u57fa\u4e8e<span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span>\u662f\u5728\u4e2d<span translate=no>_^_11_^_</span>\u5b9a\u4e49\u7684\u7c7b\u7684\u5b9e\u4f8b <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li>\u201coptimized_update\u201d \u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u5728\u6dfb\u52a0\u540e\u662f\u5426\u8981\u4f18\u5316\u7b2c\u4e8c\u4e2a\u65f6\u523b\u7684\u504f\u5dee\u6821\u6b63<span translate=no>_^_13_^_</span></li>\n<li><span translate=no>_^_14_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u6307\u793a\u662f\u4f7f\u7528 AmsGrad \u8fd8\u662f\u56de\u9000\u5230\u666e\u901a\u7684 Adam</li>\n<li><span translate=no>_^_15_^_</span>\u9884\u70ed\u6b65\u6570</li>\n<li><span translate=no>_^_16_^_</span>\u578b\u53f7\u5c3a\u5bf8\uff1b\u5373\u53d8\u538b\u5668\u4e2d\u7684\u5c3a\u5bf8\u6570</li>\n<li><span translate=no>_^_17_^_</span>\u662f\u7ec4\u503c\u7684\u9ed8\u8ba4\u5b57\u5178\u3002\u5f53\u4f60\u60f3\u6269\u5c55\u7c7b\u65f6\uff0c\u8fd9\u5f88\u6709\u7528<span translate=no>_^_18_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Plot learning rate for different warmups and model sizes</h3>\n<p><span translate=no>_^_0_^_</span></p>\n": "<h3>\u7ed8\u5236\u4e0d\u540c\u9884\u70ed\u548c\u6a21\u578b\u5927\u5c0f\u7684\u5b66\u4e60\u901f\u7387</h3>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"Noam optimizer from Attention is All You Need paper": "\u300a\u6ce8\u610f\u5c31\u662f\u4f60\u6240\u9700\u8981\u7684 Noam Optimizer\u300b\u8bba\u6587",
|
||||
"This is a tutorial/implementation of Noam optimizer. Noam optimizer has a warm-up period and then an exponentially decaying learning rate.": "\u8fd9\u662f Noam \u4f18\u5316\u5668\u7684\u6559\u7a0b/\u5b9e\u73b0\u3002Noam \u4f18\u5316\u5668\u6709\u4e00\u4e2a\u9884\u70ed\u671f\uff0c\u7136\u540e\u5b66\u4e60\u7387\u5448\u6307\u6570\u7ea7\u8870\u51cf\u3002"
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Performance testing Adam</h1>\n<span translate=no>_^_0_^_</span><p><a href=\"https://colab.research.google.com/drive/1ngowaAsADj8VdZfBifu_6L6rtjGoEeoR?usp=sharing\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u30c6\u30b9\u30c8 Adam</h1>\n<span translate=no>_^_0_^_</span><p><a href=\"https://colab.research.google.com/drive/1ngowaAsADj8VdZfBifu_6L6rtjGoEeoR?usp=sharing\"><span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"Test performance of Adam implementations": "Adam \u5b9f\u88c5\u306e\u30c6\u30b9\u30c8\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9",
|
||||
"This experiment compares performance of Adam implementations.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001Adam \u5b9f\u88c5\u306e\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u3092\u6bd4\u8f03\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Performance testing Adam</h1>\n<span translate=no>_^_0_^_</span><p><a href=\"https://colab.research.google.com/drive/1ngowaAsADj8VdZfBifu_6L6rtjGoEeoR?usp=sharing\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dc3\u0dcf\u0db0\u0db1 \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf\u0dc0 \u0d86\u0daf\u0db8\u0dca</h1>\n<span translate=no>_^_0_^_</span><p><a href=\"https://colab.research.google.com/drive/1ngowaAsADj8VdZfBifu_6L6rtjGoEeoR?usp=sharing\"><span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"Test performance of Adam implementations": "\u0d86\u0daf\u0db8\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab \u0d9a\u0dcf\u0dbb\u0dca\u0dba \u0dc3\u0dcf\u0db0\u0db1\u0dba",
|
||||
"This experiment compares performance of Adam implementations.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0daf\u0db8\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba \u0dc3\u0dcf\u0db0\u0db1\u0dba \u0dc3\u0d82\u0dc3\u0db1\u0dca\u0daf\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Performance testing Adam</h1>\n<span translate=no>_^_0_^_</span><p><a href=\"https://colab.research.google.com/drive/1ngowaAsADj8VdZfBifu_6L6rtjGoEeoR?usp=sharing\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u6027\u80fd\u6d4b\u8bd5 Adam</h1>\n<span translate=no>_^_0_^_</span><p><a href=\"https://colab.research.google.com/drive/1ngowaAsADj8VdZfBifu_6L6rtjGoEeoR?usp=sharing\"><span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"Test performance of Adam implementations": "\u6d4b\u8bd5 Adam \u5b9e\u73b0\u7684\u6027\u80fd",
|
||||
"This experiment compares performance of Adam implementations.": "\u672c\u5b9e\u9a8c\u6bd4\u8f83\u4e86 Adam \u5b9e\u73b0\u7684\u6027\u80fd\u3002"
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"<h1>Rectified Adam (RAdam) optimizer</h1>\n": "<h1>\u4fee\u6b63\u3055\u308c\u305f\u30a2\u30c0\u30e0 (RaDAM) \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h1>\n",
|
||||
"<h2>Rectified Adam Optimizer</h2>\n<p>This class extends from AMSAdam optimizer defined in <a href=\"amsadam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>\u30ec\u30af\u30c6\u30a3\u30d5\u30a1\u30a4\u30c9\u30fb\u30a2\u30c0\u30e0\u30fb\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</h2>\n<p>\u3053\u306e\u30af\u30e9\u30b9\u306f\u3001\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b AmSadam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u62e1\u5f35\u3057\u305f\u3082\u306e\u3067\u3059\u3002<a href=\"amsadam.html\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Rectified Adam</h2>\n": "<h2>\u6b63\u7fa9\u306e\u30a2\u30c0\u30e0</h2>\n",
|
||||
"<h3>Approximating <span translate=no>_^_0_^_</span></h3>\n": "<h3>\u304a\u304a\u3088\u305d\u306e\u5024 <span translate=no>_^_0_^_</span></h3>\n",
|
||||
"<h3>Calculate rectification term <span translate=no>_^_0_^_</span></h3>\n": "<h3>\u4fee\u6b63\u671f\u9593\u306e\u8a08\u7b97 <span translate=no>_^_0_^_</span></h3>\n",
|
||||
"<h3>Do the <em>RAdam</em> parameter update</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are the uncorrected first and second moments <span translate=no>_^_6_^_</span> and <span translate=no>_^_7_^_</span>; i.e. <span translate=no>_^_8_^_</span> and <span translate=no>_^_9_^_</span> without bias correction</li></ul>\n": "<h3><em>RadAM</em> \u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u66f4\u65b0\u3092\u884c\u3044\u307e\u3059</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_4_^_</span><span translate=no>_^_6_^_</span>\u672a\u88dc\u6b63\u306e\u7b2c1\u30e2\u30fc\u30e1\u30f3\u30c8\u3068\u7b2c2\u30e2\u30fc\u30e1\u30f3\u30c8\u3067\u3001<span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span><span translate=no>_^_9_^_</span>\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u306a\u3057 <span translate=no>_^_5_^_</span></li></ul>\n",
|
||||
"<h3>Exponential moving average as simple moving average</h3>\n": "<h3>\u5358\u7d14\u79fb\u52d5\u5e73\u5747\u3068\u3057\u3066\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747</h3>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_16_^_</span> whether to use sgd when the rectification term <span translate=no>_^_17_^_</span> is intractable. </li>\n<li><span translate=no>_^_18_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_19_^_</span>.</li></ul>\n": "<h3>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u521d\u671f\u5316</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u30ea\u30b9\u30c8\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u5b66\u7fd2\u7387 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>(,) <span translate=no>_^_4_^_</span> \u306e\u30bf\u30d7\u30eb\u3067\u3059 <span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span><span translate=no>_^_8_^_</span>\u307e\u305f\u306f\u305d\u308c\u306b\u57fa\u3065\u3044\u3066\u3044\u308b <span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span><span translate=no>_^_11_^_</span>\u3067\u5b9a\u7fa9\u3055\u308c\u3066\u3044\u308b\u30af\u30e9\u30b9\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u3067\u3059 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li><span translate=no>_^_13_^_</span>\u30bb\u30ab\u30f3\u30c9\u30e2\u30fc\u30e1\u30f3\u30c8\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u3092\u52a0\u7b97\u3057\u3066\u304b\u3089\u884c\u3046\u3053\u3068\u3067\u6700\u9069\u5316\u3059\u308b\u304b\u5426\u304b\u306e\u30d5\u30e9\u30b0\u3067\u3059 <span translate=no>_^_14_^_</span></li>\n<li><span translate=no>_^_15_^_</span>amsGrad\u3092\u4f7f\u7528\u3059\u308b\u304b\u3001\u30d7\u30ec\u30fc\u30f3\u306aAdam\u306b\u30d5\u30a9\u30fc\u30eb\u30d0\u30c3\u30af\u3059\u308b\u304b\u3092\u793a\u3059\u30d5\u30e9\u30b0\u3067\u3059</li>\n<li><span translate=no>_^_16_^_</span><span translate=no>_^_17_^_</span>\u4fee\u6b63\u9805\u304c\u6271\u3044\u306b\u304f\u3044\u5834\u5408\u306b sgd \u3092\u4f7f\u3046\u304b\u3069\u3046\u304b\u3002</li>\n<li><span translate=no>_^_18_^_</span>\u30b0\u30eb\u30fc\u30d7\u5024\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u8f9e\u66f8\u3067\u3059\u3002\u3053\u308c\u306f\u3001\u30af\u30e9\u30b9\u3092\u62e1\u5f35\u3059\u308b\u5834\u5408\u306b\u4fbf\u5229\u3067\u3059<span translate=no>_^_19_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Plot <span translate=no>_^_0_^_</span> against <span translate=no>_^_1_^_</span> for various <span translate=no>_^_2_^_</span></h3>\n<p><span translate=no>_^_3_^_</span></p>\n": "<h3><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3055\u307e\u3056\u307e\u306a\u30d7\u30ed\u30c3\u30c8\u5bfe\u8c61 <span translate=no>_^_2_^_</span></h3>\n<p><span translate=no>_^_3_^_</span></p>\n",
|
||||
"<h3>Rectification term</h3>\n": "<h3>\u4fee\u6b63\u671f\u9593</h3>\n",
|
||||
"<h3>Rectification</h3>\n": "<h3>\u6574\u6d41</h3>\n",
|
||||
"<h3>Scaled inverse chi-squared</h3>\n": "<h3>\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3055\u308c\u305f\u9006\u30ab\u30a4\u4e8c\u4e57</h3>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u4e0e\u3048\u3089\u308c\u305f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb\u306e\u66f4\u65b0\u30b9\u30c6\u30c3\u30d7\u3092\u5b9f\u884c\u3059\u308b</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc (\u30c6\u30f3\u30bd\u30eb) \u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u72b6\u614b\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u30b0\u30eb\u30fc\u30d7\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u5c5e\u6027\u3092\u683c\u7d0d\u3057\u307e\u3059</li>\n<li><span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u73fe\u5728\u306e\u52fe\u914d\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<p><a href=\"https://en.wikipedia.org/wiki/Scaled_inverse_chi-squared_distribution\">Scaled inverse chi-squared</a> is the distribution of squared inverse of mean of <span translate=no>_^_0_^_</span> normal distributions. <span translate=no>_^_1_^_</span> where <span translate=no>_^_2_^_</span>.</p>\n": "<p><a href=\"https://en.wikipedia.org/wiki/Scaled_inverse_chi-squared_distribution\">\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3055\u308c\u305f\u9006\u30ab\u30a4\u4e8c\u4e57\u306f</a>\u3001\u6b63\u898f\u5206\u5e03\u306e\u5e73\u5747\u306e\u4e8c\u4e57\u9006\u6570\u306e\u5206\u5e03\u3067\u3059\u3002<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3069\u3053<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> is tractable when <span translate=no>_^_1_^_</span>. We are being a little more conservative since it's an approximated value </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3069\u3093\u306a\u3068\u304d\u3067\u3082\u6271\u3044\u3084\u3059\u3044\u3002\u304a\u304a\u3088\u305d\u306e\u5024\u306a\u306e\u3067\u3001\u3082\u3046\u5c11\u3057\u4fdd\u5b88\u7684\u306b\u3057\u3066\u3044\u307e\u3059</p>\n",
|
||||
"<p>Adam optimizer sometimes converges to a bad local optima during the initial stages of the training; especially when training transformers. Researches use warmups to counter this; for the the initial training steps (warm-up stage) they use a low learning rate. This paper identifies the problem to be the high variance of adaptive learning rate during initial stages of training, and counters it using a new rectification term to reduce variance.</p>\n": "<p>\u30a2\u30c0\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306f\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u521d\u671f\u6bb5\u968e\u3001\u7279\u306b\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u3066\u3044\u308b\u3068\u304d\u306b\u3001\u4e0d\u9069\u5207\u306a\u5c40\u6240\u6700\u9069\u5024\u306b\u53ce\u675f\u3059\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u3002\u7814\u7a76\u8005\u306f\u3053\u308c\u306b\u5bfe\u6297\u3059\u308b\u305f\u3081\u306b\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u3092\u4f7f\u3044\u307e\u3059\u3002\u6700\u521d\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b9\u30c6\u30c3\u30d7\uff08\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6bb5\u968e\uff09\u3067\u306f\u4f4e\u3044\u5b66\u7fd2\u7387\u3092\u4f7f\u3044\u307e\u3059\u3002\u672c\u7a3f\u3067\u306f\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u521d\u671f\u6bb5\u968e\u306b\u304a\u3051\u308b\u9069\u5fdc\u5b66\u7fd2\u7387\u306e\u3070\u3089\u3064\u304d\u304c\u5927\u304d\u3044\u3068\u3044\u3046\u554f\u984c\u3092\u7279\u5b9a\u3057\u3001\u5206\u6563\u3092\u6e1b\u3089\u3059\u305f\u3081\u306e\u65b0\u3057\u3044\u4fee\u6b63\u9805\u3092\u7528\u3044\u3066\u305d\u306e\u554f\u984c\u306b\u5bfe\u51e6\u3057\u3066\u3044\u307e\u3059</p>\u3002\n",
|
||||
"<p>Bias correction term for <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u306e\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u7528\u8a9e <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u30b9\u30c6\u30c3\u30d7\u6570\u306e\u8a08\u7b97 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u4f53\u91cd\u6e1b\u5c11\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Computation without optimization </p>\n": "<p>\u6700\u9069\u5316\u306a\u3057\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Denominator <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5206\u6bcd <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>From <span translate=no>_^_0_^_</span> distribution we have,</p>\n": "<p><span translate=no>_^_0_^_</span>\u79c1\u305f\u3061\u304c\u6301\u3063\u3066\u3044\u308b\u30c7\u30a3\u30b9\u30c8\u30ea\u30d3\u30e5\u30fc\u30b7\u30e7\u30f3\u304b\u3089\u3001</p>\n",
|
||||
"<p>From above we have <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span>. Note that <span translate=no>_^_2_^_</span> here is the standard deviation and different from <span translate=no>_^_3_^_</span> for momentum.</p>\n": "<p>\u4e0a\u304b\u3089\u898b\u308b\u3068\u3001<span translate=no>_^_0_^_</span>\u5834\u6240\u304c\u308f\u304b\u308a\u307e\u3059<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u3053\u308c\u306f\u6a19\u6e96\u504f\u5dee\u3067\u3042\u308a\u3001<span translate=no>_^_3_^_</span>\u904b\u52d5\u91cf\u3068\u306f\u7570\u306a\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044</p>\u3002\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u53d6\u5f97\u3057\u3066 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>; i.e. <span translate=no>_^_2_^_</span> and <span translate=no>_^_3_^_</span> without bias correction </p>\n": "<p>Get <span translate=no>_^_0_^_</span> \u3068<span translate=no>_^_1_^_</span>; \u3064\u307e\u308a<span translate=no>_^_2_^_</span>\u3001<span translate=no>_^_3_^_</span>\u30d0\u30a4\u30a2\u30b9\u88dc\u6b63\u306a\u3057</p>\n",
|
||||
"<p>Get learning rate </p>\n": "<p>\u5b66\u7fd2\u7387\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Here we are taking the simple moving average of the last <span translate=no>_^_0_^_</span> gradients. <span translate=no>_^_1_^_</span> satisfies the following,</p>\n": "<p>\u3053\u3053\u3067\u306f\u3001<span translate=no>_^_0_^_</span>\u6700\u5f8c\u306e\u52fe\u914d\u306e\u5358\u7d14\u79fb\u52d5\u5e73\u5747\u3092\u53d6\u3063\u3066\u3044\u307e\u3059\u3002<span translate=no>_^_1_^_</span>\u4ee5\u4e0b\u3092\u6e80\u305f\u3057\u3001</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> is intractable </p>\n": "<p><span translate=no>_^_0_^_</span>\u6cbb\u308a\u306b\u304f\u3044\u5834\u5408</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> is intractable do a SGD with momentum </p>\n": "<p><span translate=no>_^_0_^_</span>\u624b\u306b\u8ca0\u3048\u306a\u3044\u306a\u3089\u52e2\u3044\u3092\u3064\u3051\u3066SGD\u3092\u3084\u308a\u307e\u3057\u3087\u3046</p>\n",
|
||||
"<p>In order to ensure that the adaptive learning rate <span translate=no>_^_0_^_</span> has consistent variance, we rectify the variance with <span translate=no>_^_1_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u9069\u5fdc\u578b\u5b66\u7fd2\u7387\u306e\u3070\u3089\u3064\u304d\u304c\u4e00\u8cab\u3057\u3066\u3044\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3059\u308b\u305f\u3081\u306b\u3001\u5dee\u7570\u3092\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u4fee\u6b63\u3057\u307e\u3059\u3002<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Let <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> be the functions to calculate momentum and adaptive learning rate. For Adam, they are</p>\n": "<p><span translate=no>_^_1_^_</span>\u904b\u52d5\u91cf\u3068\u9069\u5fdc\u5b66\u7fd2\u7387\u3092\u8a08\u7b97\u3059\u308b\u95a2\u6570\u3068\u3057\u307e\u3057\u3087\u3046<span translate=no>_^_0_^_</span>\u3002\u30a2\u30c0\u30e0\u306b\u3068\u3063\u3066\u3001\u5f7c\u3089\u306f</p>\n",
|
||||
"<p>Perform <em>RAdam</em> update </p>\n": "<p><em>RaDAM \u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u3092\u5b9f\u884c</em></p>\n",
|
||||
"<p>Step size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30c6\u30c3\u30d7\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The distribution of exponential moving average can be approximated as a simple moving average.</p>\n": "<p>\u6307\u6570\u79fb\u52d5\u5e73\u5747\u306e\u5206\u5e03\u306f\u3001\u5358\u7d14\u306a\u79fb\u52d5\u5e73\u5747\u3068\u3057\u3066\u8fd1\u4f3c\u3067\u304d\u307e\u3059\u3002</p>\n",
|
||||
"<p>The paper also evaluates two variance reduction mechanisms: <em> <strong>Adam-2k</strong>: Only compute the adaptive learning rate (<span translate=no>_^_0_^_</span> in <a href=\"adam.html\">Adam</a>) during the first 2k steps, without changing parameters or calculating momentum (<span translate=no>_^_1_^_</span>). </em> <strong>Adam-eps</strong>: Adam with large <span translate=no>_^_2_^_</span>.</p>\n": "<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u30012\u3064\u306e\u5206\u6563\u524a\u6e1b\u30e1\u30ab\u30cb\u30ba\u30e0\u306b\u3064\u3044\u3066\u3082\u8a55\u4fa1\u3057\u3066\u3044\u307e\u3059\u3002<em><strong>Adam-2k</strong>\uff1a\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u5909\u66f4\u3057\u305f\u308a\u3001\u904b\u52d5\u91cf\u3092\u8a08\u7b97\u3057\u305f\u308a\u305b\u305a\u306b\u3001<span translate=no>_^_0_^_</span>\u6700\u521d\u306e2k\u30b9\u30c6\u30c3\u30d7\u3067\u306f\uff08<a href=\"adam.html\">Adam\u3067</a>\uff09\u9069\u5fdc\u5b66\u7fd2\u7387\u306e\u307f\u3092\u8a08\u7b97\u3057\u307e\u3059</em>\uff08\uff09\u3002<span translate=no>_^_1_^_</span><strong>Adam-EPS</strong>: \u30a2\u30c0\u30e0\u30fb\u30a6\u30a3\u30ba\u30fb\u30e9\u30fc\u30b8\u30fb\u30a6\u30a3\u30ba\u30fb\u30e9\u30fc\u30b8</p>. <span translate=no>_^_2_^_</span>\n",
|
||||
"<p>Therefore the variance is minimized at maximal <span translate=no>_^_0_^_</span> which is <span translate=no>_^_1_^_</span>. Let the minimum variance be <span translate=no>_^_2_^_</span></p>\n": "<p>\u3057\u305f\u304c\u3063\u3066\u3001\u5206\u6563\u306f\u6700\u5927\u5024<span translate=no>_^_0_^_</span>\u3001\u3064\u307e\u308a\u3067\u6700\u5c0f\u5316\u3055\u308c\u307e\u3059\u3002<span translate=no>_^_1_^_</span>\u6700\u5c0f\u5206\u6563\u3092\u6b21\u306e\u5f0f\u306b\u3057\u307e\u3057\u3087\u3046 <span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>They estimate <span translate=no>_^_0_^_</span> based on first order expansion of <span translate=no>_^_1_^_</span> \ud83e\udd2a I didn't get how it was derived.</p>\n": "<p>\u3069\u3046\u5c0e\u304d\u51fa\u3055\u308c\u305f\u306e\u304b\u308f\u304b\u3089\u306a\u304b\u3063\u305f <span translate=no>_^_1_^_</span> \ud83e\udd2a <span translate=no>_^_0_^_</span> \u306e\u4e00\u6b21\u5c55\u958b\u306b\u57fa\u3065\u3044\u3066\u898b\u7a4d\u3082\u3063\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<p>They prove that variance of <span translate=no>_^_0_^_</span> decreases with <span translate=no>_^_1_^_</span> when <span translate=no>_^_2_^_</span>.</p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u9593\u3068\u3068\u3082\u306b\u3070\u3089\u3064\u304d\u304c\u5c0f\u3055\u304f\u306a\u308b\u3053\u3068\u3092\u8a3c\u660e\u3057\u3066\u3044\u307e\u3059<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>This gives,</p>\n": "<p>\u3053\u308c\u306b\u3088\u308a\u3001</p>\n",
|
||||
"<p>This implementation is based on <a href=\"https://github.com/LiyuanLucasLiu/RAdam\">the official implementation</a> of the paper <a href=\"https://arxiv.org/abs/1908.03265\">On the Variance of the Adaptive Learning Rate and Beyond</a>.</p>\n": "<p>\u3053\u306e\u5b9f\u88c5\u306f<a href=\"https://github.com/LiyuanLucasLiu/RAdam\">\u3001\u300c<a href=\"https://arxiv.org/abs/1908.03265\">\u9069\u5fdc\u5b66\u7fd2\u7387\u3068\u305d\u306e\u5f8c\u306e\u5dee\u7570\u306b\u95a2\u3059\u308b\u8ad6\u6587\u300d\u306e\u516c\u5f0f\u5b9f\u88c5\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a></a>\u3002</p>\n",
|
||||
"<p>Update parameters <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u66f4\u65b0 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>We have implemented it in <a href=\"https://pytorch.org\">PyTorch</a> as an extension to <a href=\"amsgrad.html\">our AMSGrad implementation</a> thus requiring only the modifications to be implemented.</p>\n": "<p><a href=\"https://pytorch.org\">amsGrad\u5b9f\u88c5\u306e\u62e1\u5f35\u3068\u3057\u3066PyTorch\u306b\u5b9f\u88c5\u3057\u305f\u306e\u3067</a><a href=\"amsgrad.html\">\u3001\u5b9f\u88c5\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b\u306e\u306f\u5909\u66f4\u3060\u3051\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>We have</p>\n": "<p>\u79c1\u305f\u3061\u306f\u6301\u3063\u3066\u3044\u307e\u3059</p>\n",
|
||||
"<p>Whether to optimize the computation by combining scalar computations </p>\n": "<p>\u30b9\u30ab\u30e9\u30fc\u8a08\u7b97\u3092\u7d44\u307f\u5408\u308f\u305b\u3066\u8a08\u7b97\u3092\u6700\u9069\u5316\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>where <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> for <span translate=no>_^_2_^_</span>. Lt <span translate=no>_^_3_^_</span> and step <span translate=no>_^_4_^_</span> be <span translate=no>_^_5_^_</span>, and <span translate=no>_^_6_^_</span> be the rectification term at step <span translate=no>_^_7_^_</span>.</p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3069\u3053\u304c<span translate=no>_^_2_^_</span>.<span translate=no>_^_3_^_</span><span translate=no>_^_4_^_</span>\u4e00\u6b69\u3092\u8e0f\u307f\u51fa\u3057\u3066<span translate=no>_^_5_^_</span>\u3001<span translate=no>_^_6_^_</span>\u6bb5\u968e\u7684\u306a\u4fee\u6b63\u9805\u306b\u306a\u308a\u306a\u3055\u3044</p>\u3002<span translate=no>_^_7_^_</span>\n",
|
||||
"<p>which gives, <span translate=no>_^_0_^_</span></p>\n": "<p>\u3053\u308c\u306b\u3088\u308a\u3001<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<span translate=no>_^_0_^_</span>": "<span translate=no>_^_0_^_</span>",
|
||||
"A simple PyTorch implementation/tutorial of RAdam optimizer.": "RaDAM \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u7c21\u5358\u306a PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"Rectified Adam (RAdam) optimizer": "\u4fee\u6b63\u3055\u308c\u305f\u30a2\u30c0\u30e0 (RaDAM) \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"<h1>Rectified Adam (RAdam) optimizer</h1>\n": "<h1>\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d86\u0daf\u0db8\u0dca (RaDAM) \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2</h1>\n",
|
||||
"<h2>Rectified Adam Optimizer</h2>\n<p>This class extends from AMSAdam optimizer defined in <a href=\"amsadam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0db8\u0dd9\u0db8\u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0d87\u0db8\u0dca\u0dc3\u0dcf\u0da9\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dd9\u0db1\u0dca \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad <a href=\"amsadam.html\"><span translate=no>_^_0_^_</span></a>. </p>\n",
|
||||
"<h2>Rectified Adam</h2>\n": "<h2>\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d86\u0daf\u0db8\u0dca</h2>\n",
|
||||
"<h3>Approximating <span translate=no>_^_0_^_</span></h3>\n": "<h3>\u0d86\u0dc3\u0db1\u0dca\u0db1\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <span translate=no>_^_0_^_</span></h3>\n",
|
||||
"<h3>Calculate rectification term <span translate=no>_^_0_^_</span></h3>\n": "<h3>\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0daf\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span></h3>\n",
|
||||
"<h3>Do the <em>RAdam</em> parameter update</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are the uncorrected first and second moments <span translate=no>_^_6_^_</span> and <span translate=no>_^_7_^_</span>; i.e. <span translate=no>_^_8_^_</span> and <span translate=no>_^_9_^_</span> without bias correction</li></ul>\n": "<h3><em>RadAM</em> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n<li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> <span translate=no>_^_5_^_</span> \u0dc3\u0dc4 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0db1\u0ddc\u0d9a\u0dc5 \u0db4\u0dc5\u0db8\u0dd4 \u0dc4\u0dcf \u0daf\u0dd9\u0dc0\u0db1 \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf <span translate=no>_^_6_^_</span> \u0dc3\u0dc4 <span translate=no>_^_7_^_</span>; i.e. <span translate=no>_^_8_^_</span> <span translate=no>_^_9_^_</span> \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2\u0db1\u0dca \u0dad\u0ddc\u0dbb\u0dc0</li></ul>\n",
|
||||
"<h3>Exponential moving average as simple moving average</h3>\n": "<h3>\u0dc3\u0dbb\u0dbd\u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0dbd\u0dd9\u0dc3 \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba</h3>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_16_^_</span> whether to use sgd when the rectification term <span translate=no>_^_17_^_</span> is intractable. </li>\n<li><span translate=no>_^_18_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_19_^_</span>.</li></ul>\n": "<h3>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0dba\u0dd2 </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dba\u0dd2 <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) \u0d9a tuple \u0dc0\u0dda </li>\n<li><span translate=no>_^_6_^_</span> <span translate=no>_^_7_^_</span> \u0dc4\u0ddd \u0db8\u0dad <span translate=no>_^_8_^_</span> \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> <span translate=no>_^_11_^_</span> \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba\u0dda \u0d85\u0dc0\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0d9a\u0dd2 <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d91\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0daf\u0dd9\u0dc0\u0db1 \u0db8\u0ddc\u0dc4\u0ddc\u0dad\u0dda \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db0\u0da2\u0dba\u0d9a\u0dd2 <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> \u0d86\u0daf\u0db8\u0dca \u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf AMSGrad \u0dc4\u0ddd \u0dc0\u0dd0\u0da7\u0dd3\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd9\u0db1 \u0db0\u0da2\u0dba\u0d9a\u0dd2 </li>\n<li><span translate=no>_^_16_^_</span> \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0daf\u0dba <span translate=no>_^_17_^_</span> \u0db1\u0ddc\u0dc3\u0dd0\u0dbd\u0d9a\u0dd2\u0dba \u0dc4\u0dd0\u0d9a\u0dd2 \u0dc0\u0dd2\u0da7 sgd \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1. </li>\n<li><span translate=no>_^_18_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc1\u0db6\u0dca\u0daf \u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dd2. \u0d94\u0db6\u0da7 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dd2\u0da7 \u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0dba\u0ddd\u0da2\u0db1\u0dc0\u0dad\u0dca <span translate=no>_^_19_^_</span>\u0dc0\u0dda. </li></ul>\n",
|
||||
"<h3>Plot <span translate=no>_^_0_^_</span> against <span translate=no>_^_1_^_</span> for various <span translate=no>_^_2_^_</span></h3>\n<p><span translate=no>_^_3_^_</span></p>\n": "<h3>\u0dc0\u0dd2\u0dc0\u0dd2\u0db0 <span translate=no>_^_1_^_</span> \u0dc3\u0db3\u0dc4\u0dcf <span translate=no>_^_0_^_</span> \u0d9a\u0dd4\u0db8\u0db1\u0dca\u0dad\u0dca\u0dbb\u0dab\u0dba <span translate=no>_^_2_^_</span></h3>\n<p><span translate=no>_^_3_^_</span></p>\n",
|
||||
"<h3>Rectification term</h3>\n": "<h3>\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0daf\u0dba</h3>\n",
|
||||
"<h3>Rectification</h3>\n": "<h3>\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8</h3>\n",
|
||||
"<h3>Scaled inverse chi-squared</h3>\n": "<h3>\u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0d9a\u0dc5 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0ddd\u0db8 \u0da0\u0dd2-\u0d9a\u0ddc\u0da7\u0dd4</h3>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u0daf\u0dd3\u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0d9a\u0dca \u0d9c\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc0\u0dda (tensor) </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab \u0d9c\u0dd4\u0dab\u0dcf\u0d82\u0d9c \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </li>\n<li><span translate=no>_^_2_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba <span translate=no>_^_3_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca \u0db5\u0dbd\u0dba \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a tensor \u0dc0\u0dda <span translate=no>_^_4_^_</span> </li>\n</ul><li><span translate=no>_^_5_^_</span> \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba tensor \u0dc0\u0dda <span translate=no>_^_6_^_</span></li>\n",
|
||||
"<p><a href=\"https://en.wikipedia.org/wiki/Scaled_inverse_chi-squared_distribution\">Scaled inverse chi-squared</a> is the distribution of squared inverse of mean of <span translate=no>_^_0_^_</span> normal distributions. <span translate=no>_^_1_^_</span> where <span translate=no>_^_2_^_</span>.</p>\n": "<p><a href=\"https://en.wikipedia.org/wiki/Scaled_inverse_chi-squared_distribution\">\u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0ddd\u0db8 \u0da0\u0dd2-\u0da0\u0dad\u0dd4\u0dbb\u0dc3\u0dca\u0dbb\u0dcf\u0d9a\u0dcf\u0dbb</a> \u0dba\u0db1\u0dd4 <span translate=no>_^_0_^_</span> \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0db6\u0dd9\u0daf\u0dcf\u0dc4\u0dd0\u0dbb\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba\u0dda \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0ddd\u0db8 \u0da0\u0dad\u0dd4\u0dbb\u0dc3\u0dca\u0dbb\u0dcf\u0d9a\u0dcf\u0dbb \u0db6\u0dd9\u0daf\u0dcf \u0dc4\u0dd0\u0dbb\u0dd3\u0db8\u0dba\u0dd2. <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dc4\u0dd9\u0daf <span translate=no>_^_2_^_</span>. </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> is tractable when <span translate=no>_^_1_^_</span>. We are being a little more conservative since it's an approximated value </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc0\u0dd2\u0da7 \u0dc3\u0ddc\u0dba\u0dcf\u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2\u0dba <span translate=no>_^_1_^_</span>. \u0d91\u0dba \u0d86\u0dc3\u0db1\u0dca\u0db1 \u0d85\u0d9c\u0dba\u0d9a\u0dca \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d85\u0db4\u0dd2 \u0dad\u0dc0 \u0da7\u0dd2\u0d9a\u0d9a\u0dca \u0d9c\u0dad\u0dcf\u0db1\u0dd4\u0d9c\u0dad\u0dd2\u0d9a \u0dc0\u0dd9\u0db8\u0dd4 </p>\n",
|
||||
"<p>Adam optimizer sometimes converges to a bad local optima during the initial stages of the training; especially when training transformers. Researches use warmups to counter this; for the the initial training steps (warm-up stage) they use a low learning rate. This paper identifies the problem to be the high variance of adaptive learning rate during initial stages of training, and counters it using a new rectification term to reduce variance.</p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0dda\u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a \u0d85\u0daf\u0dd2\u0dba\u0dbb\u0dc0\u0dbd\u0daf\u0dd3 \u0d87\u0da9\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0dc4\u0dbb \u0dc0\u0dd2\u0da7 \u0db1\u0dbb\u0d9a \u0daf\u0dda\u0dc1\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0da7 \u0d85\u0db7\u0dd2\u0dc3\u0dcf\u0dbb\u0dd3 \u0dc0\u0dda; \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dba\u0dd9\u0db1\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3. \u0db4\u0dbb\u0dca\u0dba\u0dda\u0dc2\u0dab\u0dba\u0db1\u0dca \u0db8\u0dd9\u0dba \u0db8\u0dd0\u0da9\u0db4\u0dd0\u0dc0\u0dd0\u0dad\u0dca\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2; \u0db8\u0dd6\u0dbd\u0dd2\u0d9a \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dc3\u0db3\u0dc4\u0dcf (\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dd2\u0dba) \u0d94\u0dc0\u0dd4\u0db1\u0dca \u0d85\u0da9\u0dd4 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2. \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0dda \u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a \u0d85\u0daf\u0dd2\u0dba\u0dbb\u0dc0\u0dbd\u0daf\u0dd3 \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0dd3 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0dda \u0d89\u0dc4\u0dc5 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0db8\u0dd9\u0db8 \u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0db8\u0d9f\u0dd2\u0db1\u0dca \u0d9c\u0dd0\u0da7\u0dc5\u0dd4\u0dc0 \u0dc4\u0db3\u0dd4\u0db1\u0dcf \u0d9c\u0db1\u0dca\u0db1\u0dcf \u0d85\u0dad\u0dbb \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0d85\u0da9\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dba\u0dd9\u0daf\u0dd4\u0db8\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd2\u0db1\u0dca \u0d91\u0dba \u0d9c\u0dab\u0db1\u0dca \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>Bias correction term for <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0db3\u0dc4\u0dcf\u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0daf\u0dba <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u0db6\u0dbb\u0d9a\u0dca\u0dc2\u0dba \u0dc0\u0dd3\u0db8 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Computation without optimization </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dd2\u0db1\u0dca\u0dad\u0ddc\u0dbb\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Denominator <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dd2\u0d9c\u0dab\u0dca\u0da8\u0dba\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>From <span translate=no>_^_0_^_</span> distribution we have,</p>\n": "<p><span translate=no>_^_0_^_</span> \u0db6\u0dd9\u0daf\u0dcf \u0dc4\u0dd0\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0dd2\u0da7 \u0d85\u0db4 \u0dc3\u0dad\u0dd4\u0dc0 \u0d87\u0dad,</p>\n",
|
||||
"<p>From above we have <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span>. Note that <span translate=no>_^_2_^_</span> here is the standard deviation and different from <span translate=no>_^_3_^_</span> for momentum.</p>\n": "<p>\u0d89\u0dc4\u0dc5\u0dd2\u0db1\u0dca\u0d85\u0db4\u0da7 <span translate=no>_^_0_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf \u0dad\u0dd2\u0db6\u0dda <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> \u0db8\u0dd9\u0dc4\u0dd2 \u0dc3\u0db8\u0dca\u0db8\u0dad \u0d85\u0db4\u0d9c\u0db8\u0db1\u0dba \u0dc4\u0dcf \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 <span translate=no>_^_3_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da9\u0dcf \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dbd\u0db6\u0dcf <span translate=no>_^_0_^_</span> \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>; i.e. <span translate=no>_^_2_^_</span> and <span translate=no>_^_3_^_</span> without bias correction </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 <span translate=no>_^_1_^_</span>; \u0d91\u0db1\u0db8\u0dca <span translate=no>_^_2_^_</span> \u0dc3\u0dc4 \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 <span translate=no>_^_3_^_</span> \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2\u0db1\u0dca \u0dad\u0ddc\u0dbb\u0dc0 </p>\n",
|
||||
"<p>Get learning rate </p>\n": "<p>\u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Here we are taking the simple moving average of the last <span translate=no>_^_0_^_</span> gradients. <span translate=no>_^_1_^_</span> satisfies the following,</p>\n": "<p>\u0db8\u0dd9\u0db1\u0dca\u0db1\u0d85\u0db4\u0dd2 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 <span translate=no>_^_0_^_</span> \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda \u0dc3\u0dbb\u0dbd \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0d9c\u0db1\u0dca\u0db1\u0dd9\u0db8\u0dd4. <span translate=no>_^_1_^_</span> \u0db4\u0dc4\u0dad \u0dc3\u0db3\u0dc4\u0db1\u0dca \u0daf\u0dd1 \u0dad\u0dd8\u0db4\u0dca\u0dad\u0dd2\u0db8\u0dad\u0dca \u0d9a\u0dbb\u0dba\u0dd2,</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> is intractable </p>\n": "<p>\u0d87\u0daf <span translate=no>_^_0_^_</span> \u0d9c\u0dad \u0db1\u0ddc\u0dc4\u0dd0\u0d9a\u0dd2 \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> is intractable do a SGD with momentum </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0dad \u0db1\u0ddc\u0dc4\u0dd0\u0d9a\u0dd2 <span translate=no>_^_0_^_</span> \u0db1\u0db8\u0dca \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0dc3\u0db8\u0d9c SGD \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>In order to ensure that the adaptive learning rate <span translate=no>_^_0_^_</span> has consistent variance, we rectify the variance with <span translate=no>_^_1_^_</span></p>\n": "<p>\u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0dd3\u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba\u0da7 \u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0\u0dba\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0d87\u0dad\u0dd2 \u0db6\u0dc0 \u0dc3\u0dc4\u0dad\u0dd2\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf, \u0d85\u0db4\u0dd2 \u0dc0\u0dd2\u0da0\u0dbd\u0db1\u0dba \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dbb\u0db8\u0dd4 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Let <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> be the functions to calculate momentum and adaptive learning rate. For Adam, they are</p>\n": "<p>\u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 <span translate=no>_^_1_^_</span> \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0dd3 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba\u0db1\u0dca \u0d9a\u0dbb\u0db8\u0dd4. \u0d86\u0daf\u0db8\u0dca \u0dc3\u0db3\u0dc4\u0dcf, \u0d94\u0dc0\u0dd4\u0db1\u0dca</p>\n",
|
||||
"<p>Perform <em>RAdam</em> update </p>\n": "<p><em>RadAM</em> \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dd2\u0daf\u0dd4 </p>\n",
|
||||
"<p>Step size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>The distribution of exponential moving average can be approximated as a simple moving average.</p>\n": "<p>\u0d9d\u0dcf\u0dad\u0dd3\u0dba\u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0db6\u0dd9\u0daf\u0dcf \u0dc4\u0dd0\u0dbb\u0dd3\u0db8 \u0dc3\u0dbb\u0dbd \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 \u0d86\u0dc3\u0db1\u0dca\u0db1 \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. </p>\n",
|
||||
"<p>The paper also evaluates two variance reduction mechanisms: <em> <strong>Adam-2k</strong>: Only compute the adaptive learning rate (<span translate=no>_^_0_^_</span> in <a href=\"adam.html\">Adam</a>) during the first 2k steps, without changing parameters or calculating momentum (<span translate=no>_^_1_^_</span>). </em> <strong>Adam-eps</strong>: Adam with large <span translate=no>_^_2_^_</span>.</p>\n": "<p>\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2\u0dc0\u0dd2\u0da0\u0dbd\u0dca\u0dba\u0dad\u0dcf \u0d85\u0da9\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dba\u0dcf\u0db1\u0dca\u0dad\u0dca\u0dbb\u0dab \u0daf\u0dd9\u0d9a\u0d9a\u0dca \u0daf \u0d87\u0d9c\u0dba\u0dd3\u0db8\u0da7 \u0dbd\u0d9a\u0dca \u0d9a\u0dbb\u0dba\u0dd2: <em> <strong>\u0d87\u0da9\u0db8\u0dca-2K</strong>: \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0db1\u0ddc\u0d9a\u0dbb \u0dc4\u0ddd \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0db1\u0ddc\u0d9a\u0dbb \u0db4\u0dc5\u0db8\u0dd4 2k \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dad\u0dd4\u0dc5 \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0dd3 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba (<span translate=no>_^_0_^_</span> <a href=\"adam.html\">\u0d86\u0daf\u0db8\u0dca</a>\u0dc4\u0dd2) \u0db4\u0db8\u0dab\u0d9a\u0dca \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 ( <span translate=no>_^_1_^_</span>). </em> <strong>\u0d87\u0da9\u0db8\u0dca-\u0d8a\u0db4\u0dd3\u0d91\u0dc3\u0dca</strong>: \u0d86\u0daf\u0db8\u0dca \u0dc0\u0dd2\u0dc1\u0dcf\u0dbd <span translate=no>_^_2_^_</span>. </p>\n",
|
||||
"<p>Therefore the variance is minimized at maximal <span translate=no>_^_0_^_</span> which is <span translate=no>_^_1_^_</span>. Let the minimum variance be <span translate=no>_^_2_^_</span></p>\n": "<p>\u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca\u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0d8b\u0db4\u0dbb\u0dd2\u0db8 <span translate=no>_^_0_^_</span> \u0dc0\u0dc1\u0dba\u0dd9\u0db1\u0dca \u0d85\u0dc0\u0db8 \u0d9a\u0dbb <span translate=no>_^_1_^_</span>\u0d87\u0dad. \u0d85\u0dc0\u0db8 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dc0\u0dd3\u0db8\u0da7 \u0d89\u0da9 \u0daf\u0dd9\u0db1\u0dca\u0db1 <span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>They estimate <span translate=no>_^_0_^_</span> based on first order expansion of <span translate=no>_^_1_^_</span> \ud83e\udd2a I didn't get how it was derived.</p>\n": "<p>\u0d94\u0dc0\u0dd4\u0db1\u0dca <span translate=no>_^_1_^_</span> \ud83e\udd2a \u0db4\u0dc5\u0db8\u0dd4 \u0db4\u0dd2\u0dab\u0dd2\u0dc3 \u0db4\u0dd4\u0dc5\u0dd4\u0dbd\u0dca \u0db8\u0dad <span translate=no>_^_0_^_</span> \u0db4\u0daf\u0db1\u0db8\u0dca \u0dad\u0d9a\u0dca\u0dc3\u0dda\u0dbb\u0dd4 \u0db8\u0db8 \u0d91\u0dba \u0dc0\u0dca\u0dba\u0dd4\u0dad\u0dca\u0db4\u0db1\u0dca\u0db1 \u0d9a\u0dbb\u0db1 \u0d86\u0d9a\u0dcf\u0dbb\u0dba \u0dbd\u0dd0\u0db6\u0dd4\u0dab\u0dda \u0db1\u0dd0\u0dc4\u0dd0. </p>\n",
|
||||
"<p>They prove that variance of <span translate=no>_^_0_^_</span> decreases with <span translate=no>_^_1_^_</span> when <span translate=no>_^_2_^_</span>.</p>\n": "<p>\u0d94\u0dc0\u0dd4\u0db1\u0dca\u0dc0\u0dd2\u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 <span translate=no>_^_1_^_</span> \u0dc0\u0dd2\u0da7 <span translate=no>_^_0_^_</span> \u0d85\u0da9\u0dd4 \u0dc0\u0db1 \u0db6\u0dc0 \u0d94\u0dc0\u0dd4\u0dc4\u0dd4 \u0d94\u0db4\u0dca\u0db4\u0dd4 <span translate=no>_^_2_^_</span>\u0d9a\u0dbb\u0dad\u0dd2. </p>\n",
|
||||
"<p>This gives,</p>\n": "<p>\u0db8\u0dd9\u0dba\u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0dba\u0dd2,</p>\n",
|
||||
"<p>This implementation is based on <a href=\"https://github.com/LiyuanLucasLiu/RAdam\">the official implementation</a> of the paper <a href=\"https://arxiv.org/abs/1908.03265\">On the Variance of the Adaptive Learning Rate and Beyond</a>.</p>\n": "<p>\u0db8\u0dd9\u0db8\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://github.com/LiyuanLucasLiu/RAdam\">\u0db1\u0dd2\u0dbd \u0dc0\u0dc1\u0dba\u0dd9\u0db1\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> <a href=\"https://arxiv.org/abs/1908.03265\">\u0db8\u0dad \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0dd3 \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0dc3\u0dc4 \u0d89\u0db1\u0dca \u0d94\u0db6\u0dca\u0db6\u0da7 \u0dc0\u0dd2\u0da0\u0dbd\u0dca\u0dba\u0dad\u0dcf\u0dc0</a> \u0db8\u0dad \u0dba. </p>\n",
|
||||
"<p>Update parameters <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>We have implemented it in <a href=\"https://pytorch.org\">PyTorch</a> as an extension to <a href=\"amsgrad.html\">our AMSGrad implementation</a> thus requiring only the modifications to be implemented.</p>\n": "<p><a href=\"amsgrad.html\">\u0d85\u0db4\u0d9c\u0dda AMSGrad \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0daf\u0dd2\u0d9c\u0dd4\u0dc0\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 \u0d85\u0db4\u0dd2 \u0d91\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0dc4\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a</a> \u0d9a\u0dbb \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0d91\u0db8\u0d9f\u0dd2\u0db1\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0d9a\u0db8\u0dca \u0db4\u0db8\u0dab\u0d9a\u0dca \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dda. </p>\n",
|
||||
"<p>We have</p>\n": "<p>\u0d85\u0db4\u0dc3\u0dad\u0dd4\u0dc0 \u0d87\u0dad</p>\n",
|
||||
"<p>Whether to optimize the computation by combining scalar computations </p>\n": "<p>Scalar\u0d9c\u0dab\u0db1\u0dba \u0d92\u0d9a\u0dcf\u0db6\u0daf\u0dca\u0db0 \u0dc0\u0dd2\u0dc3\u0dd2\u0db1\u0dca \u0d9c\u0dab\u0db1\u0dba \u0d8b\u0db4\u0dbb\u0dd2\u0db8 \u0db5\u0dbd \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>where <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> for <span translate=no>_^_2_^_</span>. Lt <span translate=no>_^_3_^_</span> and step <span translate=no>_^_4_^_</span> be <span translate=no>_^_5_^_</span>, and <span translate=no>_^_6_^_</span> be the rectification term at step <span translate=no>_^_7_^_</span>.</p>\n": "<p><span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> <span translate=no>_^_2_^_</span>\u0d9a\u0ddc\u0dc4\u0dda\u0daf? Lt <span translate=no>_^_3_^_</span> \u0dc3\u0dc4 \u0db4\u0dd2\u0dba\u0dc0\u0dbb <span translate=no>_^_4_^_</span> \u0dc0\u0db1\u0dca\u0db1 <span translate=no>_^_5_^_</span>, \u0dc3\u0dc4 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0dd9\u0db1\u0dca \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0daf\u0dba <span translate=no>_^_6_^_</span> \u0dc0\u0db1\u0dca\u0db1 <span translate=no>_^_7_^_</span>. </p>\n",
|
||||
"<p>which gives, <span translate=no>_^_0_^_</span></p>\n": "<p>\u0dbd\u0db6\u0dcf\u0daf\u0dd9\u0db1, <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<span translate=no>_^_0_^_</span>": "<span translate=no>_^_0_^_</span>",
|
||||
"A simple PyTorch implementation/tutorial of RAdam optimizer.": "RadAM \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dcf\u0dbb\u0d9a\u0dba\u0dda \u0dc3\u0dbb\u0dbd PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba.",
|
||||
"Rectified Adam (RAdam) optimizer": "\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d86\u0daf\u0db8\u0dca (RaDAM) \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2"
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"<h1>Rectified Adam (RAdam) optimizer</h1>\n": "<h1>\u6821\u6b63\u4e9a\u5f53 (raDAM) \u4f18\u5316\u5668</h1>\n",
|
||||
"<h2>Rectified Adam Optimizer</h2>\n<p>This class extends from AMSAdam optimizer defined in <a href=\"amsadam.html\"><span translate=no>_^_0_^_</span></a>.</p>\n": "<h2>\u7ea0\u6b63\u4e9a\u5f53\u4f18\u5316\u5668</h2>\n<p>\u8fd9\u4e2a\u7c7b\u662f\u4ece\u4e2d\u5b9a\u4e49\u7684 AmsadAM \u4f18\u5316\u5668\u6269\u5c55\u800c\u6765\u7684<a href=\"amsadam.html\"><span translate=no>_^_0_^_</span></a>\u3002</p>\n",
|
||||
"<h2>Rectified Adam</h2>\n": "<h2>\u7ea0\u6b63\u4e86\u4e9a\u5f53</h2>\n",
|
||||
"<h3>Approximating <span translate=no>_^_0_^_</span></h3>\n": "<h3>\u8fd1\u4f3c\u503c<span translate=no>_^_0_^_</span></h3>\n",
|
||||
"<h3>Calculate rectification term <span translate=no>_^_0_^_</span></h3>\n": "<h3>\u8ba1\u7b97\u6574\u6539\u671f\u9650<span translate=no>_^_0_^_</span></h3>\n",
|
||||
"<h3>Do the <em>RAdam</em> parameter update</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> and <span translate=no>_^_5_^_</span> are the uncorrected first and second moments <span translate=no>_^_6_^_</span> and <span translate=no>_^_7_^_</span>; i.e. <span translate=no>_^_8_^_</span> and <span translate=no>_^_9_^_</span> without bias correction</li></ul>\n": "<h3>\u662f\u5426\u66f4\u65b0 R <em>adAM</em> \u53c2\u6570</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u548c<span translate=no>_^_5_^_</span>\u662f\u672a\u6821\u6b63\u7684\u7b2c\u4e00\u4e2a\u548c\u7b2c\u4e8c\u4e2a\u65f6\u523b<span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span>\uff1b\u5373<span translate=no>_^_8_^_</span>\u548c<span translate=no>_^_9_^_</span>\u6ca1\u6709\u504f\u5dee\u6821\u6b63</li></ul>\n",
|
||||
"<h3>Exponential moving average as simple moving average</h3>\n": "<h3>\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u4f5c\u4e3a\u7b80\u5355\u79fb\u52a8\u5e73\u5747\u7ebf</h3>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> or <span translate=no>_^_8_^_</span> based on <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a flag whether to optimize the bias correction of the second moment by doing it after adding <span translate=no>_^_14_^_</span> </li>\n<li><span translate=no>_^_15_^_</span> is a flag indicating whether to use AMSGrad or fallback to plain Adam </li>\n<li><span translate=no>_^_16_^_</span> whether to use sgd when the rectification term <span translate=no>_^_17_^_</span> is intractable. </li>\n<li><span translate=no>_^_18_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_19_^_</span>.</li></ul>\n": "<h3>\u521d\u59cb\u5316\u4f18\u5316\u5668</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\u5217\u8868</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5b66\u4e60\u7387<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f (<span translate=no>_^_4_^_</span>,<span translate=no>_^_5_^_</span>) \u7684\u5143\u7ec4</li>\n<li><span translate=no>_^_6_^_</span>\u662f<span translate=no>_^_7_^_</span>\u6216<span translate=no>_^_8_^_</span>\u57fa\u4e8e<span translate=no>_^_9_^_</span></li>\n<li><span translate=no>_^_10_^_</span>\u662f\u5728\u4e2d<span translate=no>_^_11_^_</span>\u5b9a\u4e49\u7684\u7c7b\u7684\u5b9e\u4f8b <a href=\"index.html\"><span translate=no>_^_12_^_</span></a></li>\n<li><span translate=no>_^_13_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u662f\u5426\u5728\u6dfb\u52a0\u540e\u901a\u8fc7\u8fd9\u6837\u505a\u6765\u4f18\u5316\u7b2c\u4e8c\u4e2a\u65f6\u523b\u7684\u504f\u5dee\u6821\u6b63<span translate=no>_^_14_^_</span></li>\n<li><span translate=no>_^_15_^_</span>\u662f\u4e00\u4e2a\u6807\u5fd7\uff0c\u6307\u793a\u662f\u4f7f\u7528 AmsGrad \u8fd8\u662f\u56de\u9000\u5230\u666e\u901a\u7684 Adam</li>\n<li><span translate=no>_^_16_^_</span>\u7ea0\u6b63\u672f\u8bed<span translate=no>_^_17_^_</span>\u96be\u4ee5\u5904\u7406\u65f6\u662f\u5426\u4f7f\u7528 sgd\u3002</li>\n<li><span translate=no>_^_18_^_</span>\u662f\u7ec4\u503c\u7684\u9ed8\u8ba4\u5b57\u5178\u3002\u5f53\u4f60\u60f3\u6269\u5c55\u7c7b\u65f6\uff0c\u8fd9\u5f88\u6709\u7528<span translate=no>_^_19_^_</span>\u3002</li></ul>\n",
|
||||
"<h3>Plot <span translate=no>_^_0_^_</span> against <span translate=no>_^_1_^_</span> for various <span translate=no>_^_2_^_</span></h3>\n<p><span translate=no>_^_3_^_</span></p>\n": "<h3>\u9634\u8c0b<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5bf9\u6297\u5404\u79cd<span translate=no>_^_2_^_</span></h3>\n<p><span translate=no>_^_3_^_</span></p>\n",
|
||||
"<h3>Rectification term</h3>\n": "<h3>\u6574\u6539\u671f\u9650</h3>\n",
|
||||
"<h3>Rectification</h3>\n": "<h3>\u6574\u6539</h3>\n",
|
||||
"<h3>Scaled inverse chi-squared</h3>\n": "<h3>\u7f29\u653e\u53cd\u5411\u5361\u65b9</h3>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n": "<h3>\u5bf9\u7ed9\u5b9a\u53c2\u6570\u5f20\u91cf\u6267\u884c\u66f4\u65b0\u6b65\u9aa4</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u53c2\u6570\uff08\u5f20\u91cf\uff09\u7684\u4f18\u5316\u5668\u72b6\u6001</li>\n<li><span translate=no>_^_1_^_</span>\u5b58\u50a8\u53c2\u6570\u7ec4\u7684\u4f18\u5316\u7a0b\u5e8f\u5c5e\u6027</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53c2\u6570\u7684\u5f53\u524d\u68af<span translate=no>_^_3_^_</span>\u5ea6\u5f20\u91cf<span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u662f\u53c2\u6570\u5f20\u91cf<span translate=no>_^_6_^_</span></li></ul>\n",
|
||||
"<p><a href=\"https://en.wikipedia.org/wiki/Scaled_inverse_chi-squared_distribution\">Scaled inverse chi-squared</a> is the distribution of squared inverse of mean of <span translate=no>_^_0_^_</span> normal distributions. <span translate=no>_^_1_^_</span> where <span translate=no>_^_2_^_</span>.</p>\n": "<p><a href=\"https://en.wikipedia.org/wiki/Scaled_inverse_chi-squared_distribution\">\u7f29\u653e\u9006\u5361\u65b9</a>\u662f<span translate=no>_^_0_^_</span>\u6b63\u6001\u5206\u5e03\u5747\u503c\u7684\u9006\u5e73\u65b9\u5206\u5e03\u3002<span translate=no>_^_1_^_</span>\u5728\u54ea\u91cc<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> is tractable when <span translate=no>_^_1_^_</span>. We are being a little more conservative since it's an approximated value </p>\n": "<p><span translate=no>_^_0_^_</span>\u4ec0\u4e48\u65f6\u5019\u662f\u53ef\u4ee5\u5904\u7406<span translate=no>_^_1_^_</span>\u7684\u3002\u6211\u4eec\u7a0d\u5fae\u4fdd\u5b88\u4e00\u70b9\uff0c\u56e0\u4e3a\u5b83\u662f\u8fd1\u4f3c\u503c</p>\n",
|
||||
"<p>Adam optimizer sometimes converges to a bad local optima during the initial stages of the training; especially when training transformers. Researches use warmups to counter this; for the the initial training steps (warm-up stage) they use a low learning rate. This paper identifies the problem to be the high variance of adaptive learning rate during initial stages of training, and counters it using a new rectification term to reduce variance.</p>\n": "<p>\u5728\u8bad\u7ec3\u7684\u521d\u59cb\u9636\u6bb5\uff0cAdam optimizer \u6709\u65f6\u4f1a\u6536\u655b\u5230\u7cdf\u7cd5\u7684\u5c40\u90e8\u6700\u4f73\u503c\uff1b\u5c24\u5176\u662f\u5728\u8bad\u7ec3\u53d8\u5f62\u91d1\u521a\u65f6\u3002\u7814\u7a76\u4f7f\u7528\u70ed\u8eab\u6765\u5e94\u5bf9\u8fd9\u79cd\u60c5\u51b5\uff1b\u5bf9\u4e8e\u6700\u521d\u7684\u8bad\u7ec3\u6b65\u9aa4\uff08\u70ed\u8eab\u9636\u6bb5\uff09\uff0c\u4ed6\u4eec\u4f7f\u7528\u8f83\u4f4e\u7684\u5b66\u4e60\u7387\u3002\u672c\u6587\u5c06\u95ee\u9898\u786e\u5b9a\u4e3a\u8bad\u7ec3\u521d\u59cb\u9636\u6bb5\u81ea\u9002\u5e94\u5b66\u4e60\u7387\u7684\u9ad8\u65b9\u5dee\uff0c\u5e76\u4f7f\u7528\u65b0\u7684\u6821\u6b63\u672f\u8bed\u6765\u51cf\u5c11\u65b9\u5dee\u3002</p>\n",
|
||||
"<p>Bias correction term for <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> </p>\n": "<p>\u504f\u5dee\u6821\u6b63\u672f\u8bed<span translate=no>_^_0_^_</span>\uff0c<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p><span translate=no>_^_0_^_</span>\u8ba1\u7b97\u4f18\u5316\u5668\u6b65\u6570</p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>\u8ba1\u7b97\u4f53\u91cd\u8870\u51cf</p>\n",
|
||||
"<p>Computation without optimization </p>\n": "<p>\u65e0\u9700\u4f18\u5316\u7684\u8ba1\u7b97</p>\n",
|
||||
"<p>Denominator <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5206\u6bcd<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>From <span translate=no>_^_0_^_</span> distribution we have,</p>\n": "<p>\u4ece<span translate=no>_^_0_^_</span>\u5206\u53d1\u6765\u770b\uff0c</p>\n",
|
||||
"<p>From above we have <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span>. Note that <span translate=no>_^_2_^_</span> here is the standard deviation and different from <span translate=no>_^_3_^_</span> for momentum.</p>\n": "<p>\u4ece\u4e0a\u9762\u770b\uff0c\u6211\u4eec\u6709<span translate=no>_^_0_^_</span>\u54ea\u91cc<span translate=no>_^_1_^_</span>\u3002\u8bf7\u6ce8\u610f\uff0c<span translate=no>_^_2_^_</span>\u8fd9\u91cc\u662f\u6807\u51c6\u5dee\uff0c\u4e0e\u52a8<span translate=no>_^_3_^_</span>\u91cf\u4e0d\u540c\u3002</p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u83b7\u53d6<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>; i.e. <span translate=no>_^_2_^_</span> and <span translate=no>_^_3_^_</span> without bias correction </p>\n": "<p>Get<span translate=no>_^_0_^_</span> an<span translate=no>_^_1_^_</span> d; \u5373<span translate=no>_^_3_^_</span>\u4e0d<span translate=no>_^_2_^_</span>\u8fdb\u884c\u504f\u5dee\u6821\u6b63</p>\n",
|
||||
"<p>Get learning rate </p>\n": "<p>\u83b7\u53d6\u5b66\u4e60\u7387</p>\n",
|
||||
"<p>Here we are taking the simple moving average of the last <span translate=no>_^_0_^_</span> gradients. <span translate=no>_^_1_^_</span> satisfies the following,</p>\n": "<p>\u8fd9\u91cc\u6211\u4eec\u53d6\u6700\u540e\u4e00\u4e2a<span translate=no>_^_0_^_</span>\u68af\u5ea6\u7684\u7b80\u5355\u79fb\u52a8\u5e73\u5747\u7ebf\u3002<span translate=no>_^_1_^_</span>\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff0c</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> is intractable </p>\n": "<p>\u5982\u679c<span translate=no>_^_0_^_</span>\u662f\u68d8\u624b\u7684</p>\n",
|
||||
"<p>If <span translate=no>_^_0_^_</span> is intractable do a SGD with momentum </p>\n": "<p>\u5982\u679c<span translate=no>_^_0_^_</span>\u96be\u4ee5\u89e3\u51b3\uff0c\u90a3\u5c31\u7528\u52bf\u5934\u505a\u65b0\u52a0\u5761\u5143</p>\n",
|
||||
"<p>In order to ensure that the adaptive learning rate <span translate=no>_^_0_^_</span> has consistent variance, we rectify the variance with <span translate=no>_^_1_^_</span></p>\n": "<p>\u4e3a\u4e86\u786e\u4fdd\u81ea\u9002\u5e94\u5b66\u4e60\u7387<span translate=no>_^_0_^_</span>\u5177\u6709\u4e00\u81f4\u7684\u65b9\u5dee\uff0c\u6211\u4eec\u4f7f\u7528\u4ee5\u4e0b\u65b9\u6cd5\u6821\u6b63\u65b9\u5dee<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Let <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> be the functions to calculate momentum and adaptive learning rate. For Adam, they are</p>\n": "<p>\u8ba9<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span>\u6210\u4e3a\u8ba1\u7b97\u52a8\u91cf\u548c\u81ea\u9002\u5e94\u5b66\u4e60\u901f\u7387\u7684\u51fd\u6570\u3002\u5bf9\u4e9a\u5f53\u6765\u8bf4\uff0c\u4ed6\u4eec\u662f</p>\n",
|
||||
"<p>Perform <em>RAdam</em> update </p>\n": "<p>\u6267\u884c <em>raDAM</em> \u66f4\u65b0</p>\n",
|
||||
"<p>Step size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6b65\u957f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The distribution of exponential moving average can be approximated as a simple moving average.</p>\n": "<p>\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u7684\u5206\u5e03\u53ef\u4ee5\u8fd1\u4f3c\u4e3a\u7b80\u5355\u79fb\u52a8\u5e73\u5747\u7ebf\u3002</p>\n",
|
||||
"<p>The paper also evaluates two variance reduction mechanisms: <em> <strong>Adam-2k</strong>: Only compute the adaptive learning rate (<span translate=no>_^_0_^_</span> in <a href=\"adam.html\">Adam</a>) during the first 2k steps, without changing parameters or calculating momentum (<span translate=no>_^_1_^_</span>). </em> <strong>Adam-eps</strong>: Adam with large <span translate=no>_^_2_^_</span>.</p>\n": "<p>\u672c\u6587\u8fd8\u8bc4\u4f30\u4e86\u4e24\u79cd\u65b9\u5dee\u7f29\u51cf\u673a\u5236\uff1a<em><strong>adam-2K</strong>\uff1a\u4ec5\u8ba1\u7b97\u524d 2k \u6b65\u957f\u7684\u81ea\u9002\u5e94\u5b66\u4e60\u7387\uff08<span translate=no>_^_0_^_</span>\u5728 <a href=\"adam.html\">Adam</a> \u4e2d\uff09\uff0c\u800c\u4e0d\u66f4\u6539\u53c2\u6570\u6216\u8ba1\u7b97\u52a8\u91cf\uff08<span translate=no>_^_1_^_</span>)\u3002</em><strong>adam-eps</strong>\uff1aAdam \u5f88\u5927<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"<p>Therefore the variance is minimized at maximal <span translate=no>_^_0_^_</span> which is <span translate=no>_^_1_^_</span>. Let the minimum variance be <span translate=no>_^_2_^_</span></p>\n": "<p>\u56e0\u6b64\uff0c\u65b9\u5dee\u6700\u5c0f\u5316<span translate=no>_^_0_^_</span>\u4e3a\u6700\u5927\u503c<span translate=no>_^_1_^_</span>\u3002\u8ba9\u6700\u5c0f\u65b9\u5dee\u4e3a<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p>They estimate <span translate=no>_^_0_^_</span> based on first order expansion of <span translate=no>_^_1_^_</span> \ud83e\udd2a I didn't get how it was derived.</p>\n": "<p>\u4ed6\u4eec<span translate=no>_^_0_^_</span>\u6839\u636e\u4e00\u9636\u6269\u5f20\u4f30\u8ba1<span translate=no>_^_1_^_</span> \ud83e\udd2a \u6211\u4e0d\u660e\u767d\u5b83\u662f\u5982\u4f55\u5f97\u51fa\u7684\u3002</p>\n",
|
||||
"<p>They prove that variance of <span translate=no>_^_0_^_</span> decreases with <span translate=no>_^_1_^_</span> when <span translate=no>_^_2_^_</span>.</p>\n": "<p>\u4ed6\u4eec\u8bc1\u660e\u4e86\u968f\u65f6\u95f4\u53d8\u5316\u7684\u53d8\u5316<span translate=no>_^_0_^_</span>\u800c<span translate=no>_^_1_^_</span>\u964d\u4f4e<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"<p>This gives,</p>\n": "<p>\u8fd9\u7ed9\u4e86\uff0c</p>\n",
|
||||
"<p>This implementation is based on <a href=\"https://github.com/LiyuanLucasLiu/RAdam\">the official implementation</a> of the paper <a href=\"https://arxiv.org/abs/1908.03265\">On the Variance of the Adaptive Learning Rate and Beyond</a>.</p>\n": "<p>\u8be5\u5b9e\u65bd\u57fa\u4e8e<a href=\"https://github.com/LiyuanLucasLiu/RAdam\">\u300a<a href=\"https://arxiv.org/abs/1908.03265\">\u81ea\u9002\u5e94\u5b66\u4e60\u7387\u53ca\u4ee5\u540e\u7684\u5dee\u5f02\u300b\u4e00</a>\u6587\u7684\u6b63\u5f0f\u5b9e\u65bd</a>\u3002</p>\n",
|
||||
"<p>Update parameters <span translate=no>_^_0_^_</span> </p>\n": "<p>\u66f4\u65b0\u53c2\u6570<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>We have implemented it in <a href=\"https://pytorch.org\">PyTorch</a> as an extension to <a href=\"amsgrad.html\">our AMSGrad implementation</a> thus requiring only the modifications to be implemented.</p>\n": "<p>\u6211\u4eec\u5df2\u7ecf\u5728 <a href=\"https://pytorch.org\">PyTorch</a> \u4e2d\u5b9e\u73b0\u4e86\u5b83\uff0c\u4f5c\u4e3a<a href=\"amsgrad.html\">\u6211\u4eec\u7684 AmsGrad</a> \u5b9e\u73b0\u7684\u6269\u5c55\uff0c\u56e0\u6b64\u53ea\u9700\u8981\u5b9e\u65bd\u4fee\u6539\u5373\u53ef\u3002</p>\n",
|
||||
"<p>We have</p>\n": "<p>\u6211\u4eec\u6709</p>\n",
|
||||
"<p>Whether to optimize the computation by combining scalar computations </p>\n": "<p>\u662f\u5426\u901a\u8fc7\u7ec4\u5408\u6807\u91cf\u8ba1\u7b97\u6765\u4f18\u5316\u8ba1\u7b97</p>\n",
|
||||
"<p>where <span translate=no>_^_0_^_</span> is <span translate=no>_^_1_^_</span> for <span translate=no>_^_2_^_</span>. Lt <span translate=no>_^_3_^_</span> and step <span translate=no>_^_4_^_</span> be <span translate=no>_^_5_^_</span>, and <span translate=no>_^_6_^_</span> be the rectification term at step <span translate=no>_^_7_^_</span>.</p>\n": "<p>\u5728<span translate=no>_^_0_^_</span>\u54ea<span translate=no>_^_1_^_</span>\u91cc<span translate=no>_^_2_^_</span>\u3002Lt<span translate=no>_^_3_^_</span> and step<span translate=no>_^_4_^_</span> be<span translate=no>_^_5_^_</span>\uff0c\u7136\u540e<span translate=no>_^_6_^_</span>\u6210\u4e3a step \u7684\u6574\u6539\u671f\u9650<span translate=no>_^_7_^_</span>\u3002</p>\n",
|
||||
"<p>which gives, <span translate=no>_^_0_^_</span></p>\n": "<p>\u8fd9\u7ed9\u4e86\uff0c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<span translate=no>_^_0_^_</span>": "<span translate=no>_^_0_^_</span>",
|
||||
"A simple PyTorch implementation/tutorial of RAdam optimizer.": "\u4e00\u4e2a\u7b80\u5355\u7684 PyTorch \u5b9e\u73b0/RadAM \u4f18\u5316\u5668\u6559\u7a0b\u3002",
|
||||
"Rectified Adam (RAdam) optimizer": "\u6821\u6b63\u4e9a\u5f53 (raDAM) \u4f18\u5316\u5668"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/optimizers/index.html\">Optimizers</a></h1>\n<h2>Optimizer Implementations</h2>\n<ul><li><a href=\"https://nn.labml.ai/optimizers/adam.html\">Adam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/amsgrad.html\">AMSGrad Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/adam_warmup.html\">Adam Optimizer with warmup</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/noam.html\">Noam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/radam.html\">Rectified Adam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/ada_belief.html\">AdaBelief Optimizer</a> </li></ul>\n": "<h1><a href=\"https://nn.labml.ai/optimizers/index.html\">\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</a></h1>\n<h2>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u5b9f\u88c5</h2>\n<ul><li><a href=\"https://nn.labml.ai/optimizers/adam.html\">\u30a2\u30c0\u30e0\u30fb\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</a></li>\n<li><a href=\"https://nn.labml.ai/optimizers/amsgrad.html\">\u30de\u30b9\u30b0\u30e9\u30fc\u30c9\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</a></li>\n<li><a href=\"https://nn.labml.ai/optimizers/adam_warmup.html\">\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u6a5f\u80fd\u4ed8\u304d Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</a></li>\n<li><a href=\"https://nn.labml.ai/optimizers/noam.html\">\u30ce\u30fc\u30e0\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</a></li>\n<li><a href=\"https://nn.labml.ai/optimizers/radam.html\">\u30ec\u30af\u30c6\u30a3\u30d5\u30a1\u30a4\u30c9\u30fb\u30a2\u30c0\u30e0\u30fb\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</a></li>\n<li><a href=\"https://nn.labml.ai/optimizers/ada_belief.html\">\u30a2\u30c0\u30d6\u30ea\u30ea\u30fc\u30d5\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</a></li></ul>\n",
|
||||
"Optimizers": "\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/optimizers/index.html\">Optimizers</a></h1>\n<h2>Optimizer Implementations</h2>\n<ul><li><a href=\"https://nn.labml.ai/optimizers/adam.html\">Adam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/amsgrad.html\">AMSGrad Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/adam_warmup.html\">Adam Optimizer with warmup</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/noam.html\">Noam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/radam.html\">Rectified Adam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/ada_belief.html\">AdaBelief Optimizer</a> </li></ul>\n": "<h1><a href=\"https://nn.labml.ai/optimizers/index.html\">\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a></h1>\n<h2>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<ul><li><a href=\"https://nn.labml.ai/optimizers/adam.html\">\u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/amsgrad.html\">AMSGrad \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/adam_warmup.html\">\u0d8b\u0dab\u0dd4\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/noam.html\">\u0db1\u0dc0 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/radam.html\">\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/ada_belief.html\">ADABelief \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> </li></ul>\n",
|
||||
"Optimizers": "\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/optimizers/index.html\">Optimizers</a></h1>\n<h2>Optimizer Implementations</h2>\n<ul><li><a href=\"https://nn.labml.ai/optimizers/adam.html\">Adam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/amsgrad.html\">AMSGrad Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/adam_warmup.html\">Adam Optimizer with warmup</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/noam.html\">Noam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/radam.html\">Rectified Adam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/ada_belief.html\">AdaBelief Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/sophia.html\">Sophia-G Optimizer</a> </li></ul>\n": "<h1><a href=\"https://nn.labml.ai/optimizers/index.html\">Optimizers</a></h1>\n<h2>Optimizer Implementations</h2>\n<ul><li><a href=\"https://nn.labml.ai/optimizers/adam.html\">Adam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/amsgrad.html\">AMSGrad Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/adam_warmup.html\">Adam Optimizer with warmup</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/noam.html\">Noam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/radam.html\">Rectified Adam Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/ada_belief.html\">AdaBelief Optimizer</a> </li>\n<li><a href=\"https://nn.labml.ai/optimizers/sophia.html\">Sophia-G Optimizer</a> </li></ul>\n",
|
||||
"Optimizers": "\u4f18\u5316\u5668"
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"<h1>Sophia Optimizer</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <em>Sophia-G</em> from paper <a href=\"https://arxiv.org/abs/2305.14342\">Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training</a>. Official implementation is available at <a href=\"https://github.com/Liuhong99/Sophia\">Liuhong99/Sophia</a>.</p>\n<p>Sophia is more adaptive to heterogeneous curvatures than Adam, more resistant to non-convexity and rapid change of Hessian than Newton\u2019s method, and also uses a low-cost pre-conditioner.</p>\n<p>Sophia keeps diagonal Hessian estimates with EMA across iterations. The diagonal Hessian <span translate=no>_^_0_^_</span> is calculated every <span translate=no>_^_1_^_</span> steps.</p>\n<span translate=no>_^_2_^_</span><p>Sophia uses EMA of gradients <span translate=no>_^_3_^_</span>, only considers positive entries of the diagonal Hessian and does per-coordinate clipping to the update.</p>\n<span translate=no>_^_4_^_</span><p>where <span translate=no>_^_5_^_</span> is a very small value to prevent division by <span translate=no>_^_6_^_</span>.</p>\n<h3>Gauss-Newton-Bartlett (GNB) estimator</h3>\n<span translate=no>_^_7_^_</span><p>where <span translate=no>_^_8_^_</span> are the inputs, <span translate=no>_^_9_^_</span> is the batch size (number of inputs/tokens), <span translate=no>_^_10_^_</span> is cross entropy loss, and <span translate=no>_^_11_^_</span> are sampled from the logits <span translate=no>_^_12_^_</span>.</p>\n<p>Note that this hessian estimate is always positive and therefore we can replace <span translate=no>_^_13_^_</span> with <span translate=no>_^_14_^_</span>.</p>\n<p>Sophia with Gauss-Newton-Bartlett (GNB) estimator is <strong>Sophia-G</strong></p>\n<p>Here is an <a href=\"../transformers/basic/with_sophia.html\">experiment</a> that uses Sophia-G to train a transformer.</p>\n": "<h1>Sophia Optimizer</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <em>Sophia-G</em> from paper <a href=\"https://arxiv.org/abs/2305.14342\">Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training</a>. Official implementation is available at <a href=\"https://github.com/Liuhong99/Sophia\">Liuhong99/Sophia</a>.</p>\n<p>Sophia is more adaptive to heterogeneous curvatures than Adam, more resistant to non-convexity and rapid change of Hessian than Newton\u2019s method, and also uses a low-cost pre-conditioner.</p>\n<p>Sophia keeps diagonal Hessian estimates with EMA across iterations. The diagonal Hessian <span translate=no>_^_0_^_</span> is calculated every <span translate=no>_^_1_^_</span> steps.</p>\n<span translate=no>_^_2_^_</span><p>Sophia uses EMA of gradients <span translate=no>_^_3_^_</span>, only considers positive entries of the diagonal Hessian and does per-coordinate clipping to the update.</p>\n<span translate=no>_^_4_^_</span><p>where <span translate=no>_^_5_^_</span> is a very small value to prevent division by <span translate=no>_^_6_^_</span>.</p>\n<h3>Gauss-Newton-Bartlett (GNB) estimator</h3>\n<span translate=no>_^_7_^_</span><p>where <span translate=no>_^_8_^_</span> are the inputs, <span translate=no>_^_9_^_</span> is the batch size (number of inputs/tokens), <span translate=no>_^_10_^_</span> is cross entropy loss, and <span translate=no>_^_11_^_</span> are sampled from the logits <span translate=no>_^_12_^_</span>.</p>\n<p>Note that this hessian estimate is always positive and therefore we can replace <span translate=no>_^_13_^_</span> with <span translate=no>_^_14_^_</span>.</p>\n<p>Sophia with Gauss-Newton-Bartlett (GNB) estimator is <strong>Sophia-G</strong></p>\n<p>Here is an <a href=\"../transformers/basic/with_sophia.html\">experiment</a> that uses Sophia-G to train a transformer.</p>\n",
|
||||
"<h2>Sophia-G Optimizer</h2>\n<p>We extend the class <span translate=no>_^_0_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_1_^_</span></a> to implement the Sophia optimizer.</p>\n": "<h2>Sophia-G Optimizer</h2>\n<p>We extend the class <span translate=no>_^_0_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_1_^_</span></a> to implement the Sophia optimizer.</p>\n",
|
||||
"<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n": "<h3>Initialize a parameter state</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the parameter tensor <span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the maximum learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> </li>\n<li><span translate=no>_^_8_^_</span> is <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_14_^_</span>.</li></ul>\n": "<h3>Initialize the optimizer</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the list of parameters </li>\n<li><span translate=no>_^_1_^_</span> is the maximum learning rate <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is a tuple of (<span translate=no>_^_4_^_</span>, <span translate=no>_^_5_^_</span>) </li>\n<li><span translate=no>_^_6_^_</span> is <span translate=no>_^_7_^_</span> </li>\n<li><span translate=no>_^_8_^_</span> is <span translate=no>_^_9_^_</span> </li>\n<li><span translate=no>_^_10_^_</span> is an instance of class <span translate=no>_^_11_^_</span> defined in <a href=\"index.html\"><span translate=no>_^_12_^_</span></a> </li>\n<li><span translate=no>_^_13_^_</span> is a dictionary of default for group values. This is useful when you want to extend the class <span translate=no>_^_14_^_</span>.</li></ul>\n",
|
||||
"<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n<p>We do the following parameter update,</p>\n<span translate=no>_^_7_^_</span>": "<h3>Take an update step for a given parameter tensor</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the optimizer state of the parameter (tensor) </li>\n<li><span translate=no>_^_1_^_</span> stores optimizer attributes of the parameter group </li>\n<li><span translate=no>_^_2_^_</span> is the current gradient tensor <span translate=no>_^_3_^_</span> for the parameter <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the parameter tensor <span translate=no>_^_6_^_</span></li></ul>\n<p>We do the following parameter update,</p>\n<span translate=no>_^_7_^_</span>",
|
||||
"<h3>Update the EMA of Hessian diagonal <span translate=no>_^_0_^_</span></h3>\n<ul><li><span translate=no>_^_1_^_</span> is the number of tokens/inputs in the batch <span translate=no>_^_2_^_</span></li></ul>\n<span translate=no>_^_3_^_</span>": "<h3>Update the EMA of Hessian diagonal <span translate=no>_^_0_^_</span></h3>\n<ul><li><span translate=no>_^_1_^_</span> is the number of tokens/inputs in the batch <span translate=no>_^_2_^_</span></li></ul>\n<span translate=no>_^_3_^_</span>",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate weight decay </p>\n": "<p>Calculate weight decay </p>\n",
|
||||
"<p>Exponential moving average of Hessian diagonal, <span translate=no>_^_0_^_</span> </p>\n": "<p>Exponential moving average of Hessian diagonal, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Exponential moving average of gradients, <span translate=no>_^_0_^_</span> </p>\n": "<p>Exponential moving average of gradients, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> </p>\n": "<p>Get <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>Get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Get maximum learning rate <span translate=no>_^_0_^_</span> </p>\n": "<p>Get maximum learning rate <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get optimizer state </p>\n": "<p>Get optimizer state </p>\n",
|
||||
"<p>In-place calculation of <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n": "<p>In-place calculation of <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n": "<p>Increment <span translate=no>_^_0_^_</span> the number of optimizer steps </p>\n",
|
||||
"<p>Initialize state if empty </p>\n": "<p>Initialize state if empty </p>\n",
|
||||
"<p>Iterate through parameter groups </p>\n": "<p>Iterate through parameter groups </p>\n",
|
||||
"<p>Iterate through parameters </p>\n": "<p>Iterate through parameters </p>\n",
|
||||
"<p>Skip parameters without gradients </p>\n": "<p>Skip parameters without gradients </p>\n",
|
||||
"<p>This is the number of optimizer steps taken on the parameter, <span translate=no>_^_0_^_</span> </p>\n": "<p>This is the number of optimizer steps taken on the parameter, <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Update EMA Hessian diagonal</p>\n<span translate=no>_^_0_^_</span><p> </p>\n": "<p>Update EMA Hessian diagonal</p>\n<span translate=no>_^_0_^_</span><p> </p>\n",
|
||||
"A simple PyTorch implementation/tutorial of Sophia optimizer": "A simple PyTorch implementation/tutorial of Sophia optimizer",
|
||||
"Sophia Optimizer": "Sophia Optimizer"
|
||||
}
|
||||
Reference in New Issue
Block a user