chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Normalization Layers</h1>\n<ul><li><a href=\"batch_norm/index.html\">Batch Normalization</a> </li>\n<li><a href=\"layer_norm/index.html\">Layer Normalization</a> </li>\n<li><a href=\"instance_norm/index.html\">Instance Normalization</a> </li>\n<li><a href=\"group_norm/index.html\">Group Normalization</a> </li>\n<li><a href=\"weight_standardization/index.html\">Weight Standardization</a> </li>\n<li><a href=\"batch_channel_norm/index.html\">Batch-Channel Normalization</a> </li>\n<li><a href=\"deep_norm/index.html\">DeepNorm</a></li></ul>\n": "<h1>\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc</h1>\n<ul><li><a href=\"batch_norm/index.html\">\u30d0\u30c3\u30c1\u6b63\u898f\u5316</a></li>\n<li><a href=\"layer_norm/index.html\">\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316</a></li>\n<li><a href=\"instance_norm/index.html\">\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316</a></li>\n<li><a href=\"group_norm/index.html\">\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316</a></li>\n<li><a href=\"weight_standardization/index.html\">\u91cd\u91cf\u6a19\u6e96\u5316</a></li>\n<li><a href=\"batch_channel_norm/index.html\">\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316</a></li>\n</ul><li><a href=\"deep_norm/index.html\">\u30c7\u30a3\u30fc\u30d7\u30fb\u30ce\u30fc\u30e0</a></li>\n",
|
||||
"A set of PyTorch implementations/tutorials of normalization layers.": "\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc\u306ePyTorch\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u306e\u30bb\u30c3\u30c8\u3002",
|
||||
"Normalization Layers": "\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc"
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Normalization Layers</h1>\n<ul><li><a href=\"batch_norm/index.html\">Batch Normalization</a> </li>\n<li><a href=\"layer_norm/index.html\">Layer Normalization</a> </li>\n<li><a href=\"instance_norm/index.html\">Instance Normalization</a> </li>\n<li><a href=\"group_norm/index.html\">Group Normalization</a> </li>\n<li><a href=\"weight_standardization/index.html\">Weight Standardization</a> </li>\n<li><a href=\"batch_channel_norm/index.html\">Batch-Channel Normalization</a> </li>\n<li><a href=\"deep_norm/index.html\">DeepNorm</a></li></ul>\n": "<h1>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0dc3\u0dca\u0dae\u0dbb</h1>\n<ul><li><a href=\"batch_norm/index.html\">\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"layer_norm/index.html\">\u0dc3\u0dca\u0dae\u0dbb\u0dba \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"instance_norm/index.html\">\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"group_norm/index.html\">\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"weight_standardization/index.html\">\u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"batch_channel_norm/index.html\">\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</a> </li>\n<li><a href=\"deep_norm/index.html\">\u0d9c\u0dd0\u0db9\u0dd4\u0dbb\u0dd4 \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba</a></li></ul>\n",
|
||||
"A set of PyTorch implementations/tutorials of normalization layers.": "\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab \u0dc3\u0dca\u0dae\u0dbb\u0dc0\u0dbd PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0\u0d9a\u0dca.",
|
||||
"Normalization Layers": "\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dca\u0dae\u0dbb"
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Normalization Layers</h1>\n<ul><li><a href=\"batch_norm/index.html\">Batch Normalization</a> </li>\n<li><a href=\"layer_norm/index.html\">Layer Normalization</a> </li>\n<li><a href=\"instance_norm/index.html\">Instance Normalization</a> </li>\n<li><a href=\"group_norm/index.html\">Group Normalization</a> </li>\n<li><a href=\"weight_standardization/index.html\">Weight Standardization</a> </li>\n<li><a href=\"batch_channel_norm/index.html\">Batch-Channel Normalization</a> </li>\n<li><a href=\"deep_norm/index.html\">DeepNorm</a></li></ul>\n": "<h1>\u89c4\u8303\u5316\u5c42</h1>\n<ul><li><a href=\"batch_norm/index.html\">\u6279\u91cf\u6807\u51c6\u5316</a></li>\n<li><a href=\"layer_norm/index.html\">\u5c42\u89c4\u8303\u5316</a></li>\n<li><a href=\"instance_norm/index.html\">\u5b9e\u4f8b\u89c4\u8303\u5316</a></li>\n<li><a href=\"group_norm/index.html\">\u7fa4\u7ec4\u89c4\u8303\u5316</a></li>\n<li><a href=\"weight_standardization/index.html\">\u91cd\u91cf\u6807\u51c6\u5316</a></li>\n<li><a href=\"batch_channel_norm/index.html\">\u6279\u91cf\u4fe1\u9053\u89c4\u8303\u5316</a></li>\n<li><a href=\"deep_norm/index.html\">\u6df1\u5ea6\u89c4\u8303</a></li></ul>\n",
|
||||
"A set of PyTorch implementations/tutorials of normalization layers.": "\u4e00\u7ec4\u89c4\u8303\u5316\u5c42\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002",
|
||||
"Normalization Layers": "\u89c4\u8303\u5316\u5c42"
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"<h1>Batch-Channel Normalization</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch-Channel Normalization from the paper <a href=\"https://arxiv.org/abs/1903.10520\">Micro-Batch Training with Batch-Channel Normalization and Weight Standardization</a>. We also have an <a href=\"../weight_standardization/index.html\">annotated implementation of Weight Standardization</a>.</p>\n<p>Batch-Channel Normalization performs batch normalization followed by a channel normalization (similar to a <a href=\"../group_norm/index.html\">Group Normalization</a>. When the batch size is small a running mean and variance is used for batch normalization.</p>\n<p>Here is <a href=\"../weight_standardization/experiment.html\">the training code</a> for training a VGG network that uses weight standardization to classify CIFAR-10 data.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/weight_standardization/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316</h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/1903.10520\">\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u3068\u91cd\u307f\u6a19\u6e96\u5316\u306b\u3088\u308b\u30de\u30a4\u30af\u30ed\u30d0\u30c3\u30c1\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u300d<a href=\"https://pytorch.org\">\u306b\u3042\u308b\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u306ePyTorch\u5b9f\u88c5\u3067\u3059</a></a>\u3002\u307e\u305f\u3001<a href=\"../weight_standardization/index.html\">\u91cd\u91cf\u6a19\u6e96\u5316\u306e\u6ce8\u91c8\u4ed8\u304d\u5b9f\u88c5\u3082\u3042\u308a\u307e\u3059\u3002</a></p>\n<p><a href=\"../group_norm/index.html\">\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u306f\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306e\u5f8c\u306b\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u3092\u884c\u3044\u307e\u3059 (\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u3068\u540c\u69d8)\u3002</a>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u304c\u5c0f\u3055\u3044\u5834\u5408\u306f\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306b\u5b9f\u884c\u5e73\u5747\u3068\u5206\u6563\u304c\u4f7f\u7528\u3055\u308c\u307e\u3059</p>\u3002\n<p>\u91cd\u307f\u6a19\u6e96\u5316\u3092\u4f7f\u7528\u3057\u3066 CIFAR-10 \u30c7\u30fc\u30bf\u3092\u5206\u985e\u3059\u308b VGG <a href=\"../weight_standardization/experiment.html\">\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3092\u6b21\u306b\u793a\u3057\u307e\u3059</a>\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/weight_standardization/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Batch-Channel Normalization</h2>\n<p>This first performs a batch normalization - either <a href=\"../batch_norm/index.html\">normal batch norm</a> or a batch norm with estimated mean and variance (exponential mean/variance over multiple batches). Then a channel normalization performed.</p>\n": "<h2>\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316</h2>\n<p>\u3053\u308c\u306f\u6700\u521d\u306b\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092\u884c\u3044\u307e\u3059\u3002<a href=\"../batch_norm/index.html\">\u901a\u5e38\u306e\u30d0\u30c3\u30c1\u30ce\u30eb\u30e0\u304b</a>\u3001\u63a8\u5b9a\u5e73\u5747\u3068\u5206\u6563 (\u8907\u6570\u306e\u30d0\u30c3\u30c1\u306b\u308f\u305f\u308b\u6307\u6570\u95a2\u6570\u7684\u5e73\u5747/\u5206\u6563) \u3092\u542b\u3080\u30d0\u30c3\u30c1\u30ce\u30eb\u30e0\u306e\u3069\u3061\u3089\u304b\u3067\u3059\u3002\u6b21\u306b\u3001\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u304c\u5b9f\u884c\u3055\u308c\u307e\u3057\u305f</p>\u3002\n",
|
||||
"<h2>Channel Normalization</h2>\n<p>This is similar to <a href=\"../group_norm/index.html\">Group Normalization</a> but affine transform is done group wise.</p>\n": "<h2>\u30c1\u30e3\u30f3\u30cd\u30eb\u6b63\u898f\u5316</h2>\n<p><a href=\"../group_norm/index.html\">\u3053\u308c\u306f\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u306b\u4f3c\u3066\u3044\u307e\u3059\u304c</a>\u3001\u30a2\u30d5\u30a3\u30f3\u5909\u63db\u306f\u30b0\u30eb\u30fc\u30d7\u3054\u3068\u306b\u884c\u308f\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<h2>Estimated Batch Normalization</h2>\n<p>When input <span translate=no>_^_0_^_</span> is a batch of image representations, where <span translate=no>_^_1_^_</span> is the batch size, <span translate=no>_^_2_^_</span> is the number of channels, <span translate=no>_^_3_^_</span> is the height and <span translate=no>_^_4_^_</span> is the width. <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>.</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>where,</p>\n<span translate=no>_^_8_^_</span><p>are the running mean and variances. <span translate=no>_^_9_^_</span> is the momentum for calculating the exponential mean.</p>\n": "<h2>\u63a8\u5b9a\u30d0\u30c3\u30c1\u6b63\u898f\u5316</h2>\n<p><span translate=no>_^_0_^_</span>\u5165\u529b\u304c\u30a4\u30e1\u30fc\u30b8\u8868\u73fe\u306e\u30d0\u30c3\u30c1\u306e\u5834\u5408\u3001<span translate=no>_^_1_^_</span>\u306f\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3001<span translate=no>_^_2_^_</span>\u306f\u30c1\u30e3\u30cd\u30eb\u6570\u3001<span translate=no>_^_3_^_</span>\u306f\u9ad8\u3055\u3001<span translate=no>_^_4_^_</span>\u306f\u5e45\u3067\u3059\u3002<span translate=no>_^_5_^_</span>\u3068<span translate=no>_^_6_^_</span>\u3002</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>\u3069\u3053\u3001</p>\n<span translate=no>_^_8_^_</span><p>\u306f\u79fb\u52d5\u5e73\u5747\u3068\u5206\u6563\u3067\u3059\u3002<span translate=no>_^_9_^_</span>\u6307\u6570\u5e73\u5747\u3092\u8a08\u7b97\u3059\u308b\u305f\u3081\u306e\u904b\u52d5\u91cf\u3067\u3059\u3002</p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5f62\u72b6\u306e\u30c6\u30f3\u30bd\u30eb\u3067\u3059\u3002<span translate=no>_^_2_^_</span>\u4efb\u610f\u306e\u6570 (0 \u306e\u5834\u5408\u3082\u3042\u308a\u307e\u3059) \u306e\u6b21\u5143\u3092\u793a\u3057\u307e\u3059\u3002\u305f\u3068\u3048\u3070\u3001\u753b\u50cf (2D) \u306e\u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u3067\u306f\u3001\u6b21\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059</p>\u3002<span translate=no>_^_3_^_</span>\n",
|
||||
"<p>Calculate the mean across first and last dimensions; <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u521d\u306e\u6b21\u5143\u3068\u6700\u5f8c\u306e\u6b21\u5143\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the mean across last dimension; i.e. the means for each sample and channel group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u5f8c\u306e\u6b21\u5143\u306e\u5e73\u5747\u3001\u3064\u307e\u308a\u5404\u30b5\u30f3\u30d7\u30eb\u3068\u30c1\u30e3\u30cd\u30eb\u30b0\u30eb\u30fc\u30d7\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimensions; <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u521d\u306e\u6b21\u5143\u3068\u6700\u5f8c\u306e\u6b21\u5143\u306e\u4e8c\u4e57\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean across last dimension; i.e. the means for each sample and channel group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u5f8c\u306e\u6b21\u5143\u306e\u4e8c\u4e57\u5e73\u5747\u3001\u3064\u307e\u308a\u5404\u30b5\u30f3\u30d7\u30eb\u3068\u30c1\u30e3\u30cd\u30eb\u30b0\u30eb\u30fc\u30d7\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Channel normalization </p>\n": "<p>\u30c1\u30e3\u30f3\u30cd\u30eb\u6b63\u898f\u5316</p>\n",
|
||||
"<p>Channel wise transformation parameters </p>\n": "<p>\u30c1\u30e3\u30f3\u30cd\u30eb\u5358\u4f4d\u306e\u5909\u63db\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc</p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Keep old shape </p>\n": "<p>\u53e4\u3044\u5f62\u3092\u4fdd\u3064</p>\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u5143\u306e\u5f62\u3092\u4fdd\u3064</p>\n",
|
||||
"<p>No backpropagation through <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u304a\u3088\u3073\u3092\u4ecb\u3057\u305f\u30d0\u30c3\u30af\u30d7\u30ed\u30d1\u30b2\u30fc\u30b7\u30e7\u30f3\u306a\u3057 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30ce\u30fc\u30de\u30e9\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Parameters for affine transformation.</p>\n<p><em>Note that these transforms are per group, unlike in group norm where they are transformed channel-wise.</em> </p>\n": "<p>\u30a2\u30d5\u30a3\u30f3\u5909\u63db\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3002</p>\n<p><em>\u3053\u308c\u3089\u306e\u5909\u63db\u306f\u3001\u30c1\u30e3\u30cd\u30eb\u3054\u3068\u306b\u5909\u63db\u3055\u308c\u308b\u30b0\u30eb\u30fc\u30d7\u30ce\u30eb\u30e0\u3068\u306f\u7570\u306a\u308a\u3001\u30b0\u30eb\u30fc\u30d7\u3054\u3068\u306b\u884c\u308f\u308c\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002</em></p>\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5f62\u3092\u5909\u3048\u3066 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u5143\u306e\u5f62\u306b\u623b\u3057\u3066\u623b\u3059</p>\n",
|
||||
"<p>Sanity check to make sure the number of features is correct </p>\n": "<p>\u6a5f\u80fd\u306e\u6570\u304c\u6b63\u3057\u3044\u3053\u3068\u3092\u78ba\u8a8d\u3059\u308b\u305f\u3081\u306e\u30b5\u30cb\u30c6\u30a3\u30c1\u30a7\u30c3\u30af</p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u6a5f\u80fd\u306e\u6570\u304c\u540c\u3058\u3067\u3042\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3059\u308b\u305f\u3081\u306e\u30b5\u30cb\u30c6\u30a3\u30c1\u30a7\u30c3\u30af</p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30b1\u30fc\u30eb\u3068\u30b7\u30d5\u30c8 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Scale and shift group-wise <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b0\u30eb\u30fc\u30d7\u3054\u3068\u306e\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3068\u30b7\u30d5\u30c8 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Tensors for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u304a\u3088\u3073\u306e\u30c6\u30f3\u30bd\u30eb <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Update <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in training mode only </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u30a2\u30c3\u30d7\u30c7\u30fc\u30c8\u304a\u3088\u3073\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30e2\u30fc\u30c9\u306e\u307f</p>\n",
|
||||
"<p>Update exponential moving averages</p>\n<span translate=no>_^_0_^_</span><p> </p>\n": "<p>\u6307\u6570\u79fb\u52d5\u5e73\u5747\u306e\u66f4\u65b0</p>\n<span translate=no>_^_0_^_</span><p></p>\n",
|
||||
"<p>Use estimated batch norm or normal batch norm. </p>\n": "<p>\u63a8\u5b9a\u30d0\u30c3\u30c1\u30ce\u30eb\u30e0\u307e\u305f\u306f\u901a\u5e38\u306e\u30d0\u30c3\u30c1\u30ce\u30eb\u30e0\u3092\u4f7f\u7528\u3057\u3066\u304f\u3060\u3055\u3044\u3002</p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5404\u6a5f\u80fd\u306e\u5dee\u7570 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Variance for each sample and feature group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5404\u30b5\u30f3\u30d7\u30eb\u3068\u6a5f\u80fd\u30b0\u30eb\u30fc\u30d7\u306e\u5dee\u7570 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_5_^_</span> is whether to use running mean and variance for batch norm</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u5165\u529b\u5185\u306e\u7279\u5fb4\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span><span translate=no>_^_3_^_</span>\u6570\u5024\u306e\u5b89\u5b9a\u6027\u306e\u305f\u3081\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3092\u53d6\u308b\u3068\u304d\u306e\u52e2\u3044\u3067\u3059</li>\n<li><span translate=no>_^_5_^_</span>\u30d0\u30c3\u30c1\u30ce\u30eb\u30e0\u306b\u5b9f\u884c\u5e73\u5747\u3068\u5206\u6563\u3092\u4f7f\u7528\u3059\u308b\u304b\u3069\u3046\u304b\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is the number of groups the features are divided into </li>\n<li><span translate=no>_^_2_^_</span> is <span translate=no>_^_3_^_</span>, used in <span translate=no>_^_4_^_</span> for numerical stability </li>\n<li><span translate=no>_^_5_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_6_^_</span> is whether to use running mean and variance for batch norm</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u5165\u529b\u5185\u306e\u7279\u5fb4\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30d5\u30a3\u30fc\u30c1\u30e3\u304c\u5206\u5272\u3055\u308c\u3066\u3044\u308b\u30b0\u30eb\u30fc\u30d7\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span><span translate=no>_^_4_^_</span>\u6570\u5024\u306e\u5b89\u5b9a\u6027\u306e\u305f\u3081\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059 <span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3092\u53d6\u308b\u3068\u304d\u306e\u52e2\u3044\u3067\u3059</li>\n<li><span translate=no>_^_6_^_</span>\u30d0\u30c3\u30c1\u30ce\u30eb\u30e0\u306b\u5b9f\u884c\u5e73\u5747\u3068\u5206\u6563\u3092\u4f7f\u7528\u3059\u308b\u304b\u3069\u3046\u304b\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of groups the features are divided into </li>\n<li><span translate=no>_^_1_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_2_^_</span> is <span translate=no>_^_3_^_</span>, used in <span translate=no>_^_4_^_</span> for numerical stability </li>\n<li><span translate=no>_^_5_^_</span> is whether to scale and shift the normalized value</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30d5\u30a3\u30fc\u30c1\u30e3\u304c\u5206\u5272\u3055\u308c\u3066\u3044\u308b\u30b0\u30eb\u30fc\u30d7\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u5165\u529b\u5185\u306e\u7279\u5fb4\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span><span translate=no>_^_4_^_</span>\u6570\u5024\u306e\u5b89\u5b9a\u6027\u306e\u305f\u3081\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059 <span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u6b63\u898f\u5316\u3055\u308c\u305f\u5024\u3092\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3057\u3066\u30b7\u30d5\u30c8\u3059\u308b\u304b\u3069\u3046\u304b\u3067\u3059</li></ul>\n",
|
||||
"A PyTorch implementation/tutorial of Batch-Channel Normalization.": "\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u306ePyTorch\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"Batch-Channel Normalization": "\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316"
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"<h1>Batch-Channel Normalization</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch-Channel Normalization from the paper <a href=\"https://arxiv.org/abs/1903.10520\">Micro-Batch Training with Batch-Channel Normalization and Weight Standardization</a>. We also have an <a href=\"../weight_standardization/index.html\">annotated implementation of Weight Standardization</a>.</p>\n<p>Batch-Channel Normalization performs batch normalization followed by a channel normalization (similar to a <a href=\"../group_norm/index.html\">Group Normalization</a>. When the batch size is small a running mean and variance is used for batch normalization.</p>\n<p>Here is <a href=\"../weight_standardization/experiment.html\">the training code</a> for training a VGG network that uses weight standardization to classify CIFAR-10 data.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/weight_standardization/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/f4a783a2a7df11eb921d0242ac1c0002\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba <a href=\"https://arxiv.org/abs/1903.10520\">\u0dc3\u0dc4 \u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0db8\u0dba\u0dd2\u0d9a\u0dca\u0dbb\u0ddd-\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc0\u0dd9\u0dad\u0dd2\u0db1\u0dca \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</a> . <a href=\"../weight_standardization/index.html\">\u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc0\u0dd2\u0d9a\u0dd8\u0dad \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dca</a>\u0daf \u0d85\u0db4 \u0dc3\u0dad\u0dd4\u0dc0 \u0d87\u0dad. </p>\n<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dbb \u0db4\u0dc3\u0dd4\u0dc0 \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba ( <a href=\"../group_norm/index.html\">\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0da7</a>\u0dc3\u0db8\u0dcf\u0db1 \u0dc0\u0dda. \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dd4\u0da9\u0dcf \u0dc0\u0db1 \u0dc0\u0dd2\u0da7 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db0\u0dcf\u0dc0\u0db1 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba\u0d9a\u0dca \u0dc3\u0dc4 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2. </p>\n<p>CIFA-10\u0daf\u0dad\u0dca\u0dad \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 VGG \u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 <a href=\"../weight_standardization/experiment.html\">\u0d9a\u0dda\u0dad\u0dba</a> \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/weight_standardization/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/f4a783a2a7df11eb921d0242ac1c0002\"> <span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<h2>Batch-Channel Normalization</h2>\n<p>This first performs a batch normalization - either <a href=\"../batch_norm/index.html\">normal batch norm</a> or a batch norm with estimated mean and variance (exponential mean/variance over multiple batches). Then a channel normalization performed.</p>\n": "<h2>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0db8\u0dd9\u0dba\u0db8\u0dd4\u0dbd\u0dd2\u0db1\u0dca\u0db8 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2 - \u0d91\u0d9a\u0dca\u0d9a\u0ddd <a href=\"../batch_norm/index.html\">\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba</a> \u0dc4\u0ddd \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0d9c\u0dad \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dc3\u0dc4\u0dd2\u0dad \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db1\u0dd2\u0dba\u0db8\u0dba\u0d9a\u0dca (\u0db6\u0dc4\u0dd4 \u0d9a\u0dcf\u0dab\u0dca\u0da9\u0dc0\u0dbd\u0da7 \u0dc0\u0da9\u0dcf on \u0dcf\u0dad\u0dd3\u0dba \u0d85\u0dbb\u0dca\u0dae\u0dba/\u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0). \u0d91\u0dc0\u0dd2\u0da7 \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0db1 \u0dbd\u0daf\u0dd3. </p>\n",
|
||||
"<h2>Channel Normalization</h2>\n<p>This is similar to <a href=\"../group_norm/index.html\">Group Normalization</a> but affine transform is done group wise.</p>\n": "<h2>\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p>\u0db8\u0dd9\u0dba <a href=\"../group_norm/index.html\">\u0dc3\u0db8\u0dd6\u0dc4 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0da7</a> \u0dc3\u0db8\u0dcf\u0db1 \u0dc0\u0db1 \u0db1\u0db8\u0dd4\u0dad\u0dca \u0d87\u0dc6\u0dba\u0dd2\u0db1\u0dca \u0db4\u0dbb\u0dd2\u0dab\u0dcf\u0db8\u0db1\u0dba \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0dbd\u0dd9\u0dc3 \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dd9\u0dbb\u0dda. </p>\n",
|
||||
"<h2>Estimated Batch Normalization</h2>\n<p>When input <span translate=no>_^_0_^_</span> is a batch of image representations, where <span translate=no>_^_1_^_</span> is the batch size, <span translate=no>_^_2_^_</span> is the number of channels, <span translate=no>_^_3_^_</span> is the height and <span translate=no>_^_4_^_</span> is the width. <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>.</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>where,</p>\n<span translate=no>_^_8_^_</span><p>are the running mean and variances. <span translate=no>_^_9_^_</span> is the momentum for calculating the exponential mean.</p>\n": "<h2>\u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0d9c\u0dad\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</h2>\n<p><span translate=no>_^_0_^_</span> \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dbb\u0dd6\u0db4 \u0db1\u0dd2\u0dbb\u0dd6\u0db4\u0dab \u0dc3\u0db8\u0dd6\u0dc4\u0dba\u0d9a\u0dca <span translate=no>_^_1_^_</span> \u0dc0\u0db1 \u0dc0\u0dd2\u0da7, \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf, <span translate=no>_^_2_^_</span> \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0d9c\u0dab\u0db1, <span translate=no>_^_3_^_</span> \u0d8b\u0dc3 \u0dc3\u0dc4 <span translate=no>_^_4_^_</span> \u0db4\u0dc5\u0dbd \u0dc0\u0dda. <span translate=no>_^_5_^_</span> \u0dc3\u0dc4 <span translate=no>_^_6_^_</span>. </p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>\u0d9a\u0ddc\u0dc4\u0dd9\u0daf,</p>\n<span translate=no>_^_8_^_</span><p>\u0db0\u0dcf\u0dc0\u0db1\u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0dc3\u0dc4 \u0dc0\u0dd2\u0da0\u0dbd\u0dca\u0dba\u0dba\u0db1\u0dca \u0dc0\u0dda. <span translate=no>_^_9_^_</span> \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0dc0\u0dda. </p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p> <span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d86\u0dad\u0db1\u0dca\u0dba <span translate=no>_^_1_^_</span>\u0dc0\u0dda. <span translate=no>_^_2_^_</span> \u0d95\u0db1\u0dd1\u0db8 \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0\u0d9a\u0dca (\u0dc3\u0db8\u0dc4\u0dbb\u0dc0\u0dd2\u0da7 0) \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0daf\u0d9a\u0dca\u0dc0\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3, \u0dbb\u0dd6\u0db4\u0dba\u0d9a\u0dca (2D) \u0dc3\u0d82\u0d9a\u0ddd\u0da0\u0db1\u0dba \u0dad\u0dd4\u0dc5 \u0db8\u0dd9\u0dba \u0dc0\u0db1\u0dd4 \u0d87\u0dad <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p>Calculate the mean across first and last dimensions; <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0dc4\u0dcf \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc4\u0dbb\u0dc4\u0dcf \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1; <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate the mean across last dimension; i.e. the means for each sample and channel group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0db8\u0dcf\u0db1\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1; \u0d91\u0db1\u0db8\u0dca \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba \u0dc3\u0dc4 \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dcf\u0db0\u0dca\u0dba\u0dba\u0db1\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimensions; <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0dc4\u0dcf \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc4\u0dbb\u0dc4\u0dcf \u0da0\u0dad\u0dd4\u0dbb\u0dc3\u0dca\u0dbb\u0dcf\u0d9a\u0dcf\u0dbb \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1; <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate the squared mean across last dimension; i.e. the means for each sample and channel group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0db8\u0dcf\u0db1\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0dc0\u0dbb\u0dca\u0d9c \u0d9a\u0dc5 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1; \u0d91\u0db1\u0db8\u0dca \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba \u0dc3\u0dc4 \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dcf\u0db0\u0dca\u0dba\u0dba\u0db1\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Channel normalization </p>\n": "<p>\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba </p>\n",
|
||||
"<p>Channel wise transformation parameters </p>\n": "<p>\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca </p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Keep old shape </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dc4\u0dd0\u0da9\u0dba \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u0db8\u0dd4\u0dbd\u0dca\u0dc4\u0dd0\u0da9\u0dba \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>No backpropagation through <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc4\u0dbb\u0dc4\u0dcf <span translate=no>_^_0_^_</span> \u0db4\u0dc3\u0dd4\u0d9c\u0dcf\u0db8\u0dd3 \u0db4\u0dca\u0dbb\u0da0\u0dcf\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u200d\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Parameters for affine transformation.</p>\n<p><em>Note that these transforms are per group, unlike in group norm where they are transformed channel-wise.</em> </p>\n": "<p>\u0d87\u0dc6\u0dba\u0dd2\u0db1\u0dca\u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca. </p>\n<p><em>\u0db8\u0dd9\u0db8\u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0db1\u0dca \u0d91\u0d9a\u0dca \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0d9a\u0da7 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1, \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba\u0dda \u0db8\u0dd9\u0db1\u0dca \u0db1\u0ddc\u0dc0, \u0d92\u0dc0\u0dcf \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0d85\u0db1\u0dd4\u0dc0 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0dc0\u0dda. </em> </p>\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dd0\u0dc0\u0dad\u0dc4\u0dd0\u0da9\u0d9c\u0dc3\u0dca\u0dc0\u0dcf \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u0db8\u0dd4\u0dbd\u0dca\u0db4\u0dd2\u0da7\u0db4\u0dad\u0da7 \u0db1\u0dd0\u0dc0\u0dad \u0dc4\u0dd0\u0da9\u0d9c\u0dc3\u0dca\u0dc0\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0db4\u0dd0\u0db8\u0dd2\u0dab\u0dd3\u0db8 </p>\n",
|
||||
"<p>Sanity check to make sure the number of features is correct </p>\n": "<p>\u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c\u0d9c\u0dab\u0db1 \u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2 \u0db6\u0dc0 \u0dad\u0dc4\u0dc0\u0dd4\u0dbb\u0dd4 \u0d9a\u0dbb \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0db1\u0dd3\u0db4\u0dcf\u0dbb\u0d9a\u0dca\u0dc2\u0dcf\u0dc0 \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c\u0d9c\u0dab\u0db1 \u0dc3\u0db8\u0dcf\u0db1 \u0db6\u0dc0 \u0dad\u0dc4\u0dc0\u0dd4\u0dbb\u0dd4 \u0d9a\u0dbb \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0db1\u0dd3\u0db4\u0dcf\u0dbb\u0d9a\u0dca\u0dc2\u0dcf\u0dc0 \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0dc3\u0dc4 \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Scale and shift group-wise <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0d85\u0db1\u0dd4\u0dc0 \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0dc3\u0dc4 \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Tensors for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0db3\u0dc4\u0dcf\u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dca\u0dc3\u0dca <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Update <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in training mode only </p>\n": "<p>\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 <span translate=no>_^_1_^_</span> \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dca\u0dbb\u0d9a\u0dcf\u0dbb\u0dba\u0dda\u0daf\u0dd3 \u0db4\u0db8\u0dab\u0dd2 </p>\n",
|
||||
"<p>Update exponential moving averages</p>\n<span translate=no>_^_0_^_</span><p> </p>\n": "<p>\u0d9d\u0dcf\u0dad\u0dd3\u0dba\u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1</p>\n<span translate=no>_^_0_^_</span><p> </p>\n",
|
||||
"<p>Use estimated batch norm or normal batch norm. </p>\n": "<p>\u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0d9c\u0dad\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba \u0dc4\u0ddd \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dbd\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Variance for each sample and feature group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba \u0dc3\u0dc4 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_5_^_</span> is whether to use running mean and variance for batch norm</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0d86\u0daf\u0dcf\u0db1\u0dba\u0dda \u0d87\u0dad\u0dd2 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_1_^_</span> \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc3\u0dca\u0dae\u0dcf\u0dba\u0dd2\u0dad\u0dcf\u0dc0 <span translate=no>_^_3_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0dc0\u0dda <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_4_^_</span> \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0d9c\u0db1\u0dd2\u0db8\u0dd2\u0db1\u0dca \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0dc0\u0dda </li>\n<li><span translate=no>_^_5_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db0\u0dcf\u0dc0\u0db1 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0dc3\u0dc4 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1\u0dba\u0dd2</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is the number of groups the features are divided into </li>\n<li><span translate=no>_^_2_^_</span> is <span translate=no>_^_3_^_</span>, used in <span translate=no>_^_4_^_</span> for numerical stability </li>\n<li><span translate=no>_^_5_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_6_^_</span> is whether to use running mean and variance for batch norm</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0d86\u0daf\u0dcf\u0db1\u0dba\u0dda \u0d87\u0dad\u0dd2 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d9c\u0dab\u0db1 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0dc0\u0dbd\u0da7 \u0db6\u0dd9\u0daf\u0dcf \u0d87\u0dad </li>\n<li><span translate=no>_^_2_^_</span> \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc3\u0dca\u0dae\u0dcf\u0dba\u0dd2\u0dad\u0dcf\u0dc0 <span translate=no>_^_4_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0dc0\u0dda <span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_5_^_</span> \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0d9c\u0db1\u0dd2\u0db8\u0dd2\u0db1\u0dca \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0dc0\u0dda </li>\n<li><span translate=no>_^_6_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db0\u0dcf\u0dc0\u0db1 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0dc3\u0dc4 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1\u0dba\u0dd2</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of groups the features are divided into </li>\n<li><span translate=no>_^_1_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_2_^_</span> is <span translate=no>_^_3_^_</span>, used in <span translate=no>_^_4_^_</span> for numerical stability </li>\n<li><span translate=no>_^_5_^_</span> is whether to scale and shift the normalized value</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d9c\u0dab\u0db1 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0dc0\u0dbd\u0da7 \u0db6\u0dd9\u0daf\u0dcf \u0d87\u0dad </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d86\u0daf\u0dcf\u0db1\u0dba\u0dda \u0d87\u0dad\u0dd2 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_2_^_</span> \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc3\u0dca\u0dae\u0dcf\u0dba\u0dd2\u0dad\u0dcf\u0dc0 <span translate=no>_^_4_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0dc0\u0dda <span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_5_^_</span> \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dc5 \u0d85\u0d9c\u0dba \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1\u0dba\u0dd2</li></ul>\n",
|
||||
"A PyTorch implementation/tutorial of Batch-Channel Normalization.": "\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba.",
|
||||
"Batch-Channel Normalization": "\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"<h1>Batch-Channel Normalization</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch-Channel Normalization from the paper <a href=\"https://arxiv.org/abs/1903.10520\">Micro-Batch Training with Batch-Channel Normalization and Weight Standardization</a>. We also have an <a href=\"../weight_standardization/index.html\">annotated implementation of Weight Standardization</a>.</p>\n<p>Batch-Channel Normalization performs batch normalization followed by a channel normalization (similar to a <a href=\"../group_norm/index.html\">Group Normalization</a>. When the batch size is small a running mean and variance is used for batch normalization.</p>\n<p>Here is <a href=\"../weight_standardization/experiment.html\">the training code</a> for training a VGG network that uses weight standardization to classify CIFAR-10 data.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/weight_standardization/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u6279\u5904\u7406\u4fe1\u9053\u6807\u51c6\u5316</h1>\n<p>\u8fd9\u662f <a href=\"https://pytorch.org\">PyTorch</a> \u5b9e\u73b0\u7684\u6279\u5904\u7406\u901a\u9053\u6807\u51c6\u5316\uff0c\u6765\u81ea\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/1903.10520\">\u4f7f\u7528\u6279\u5904\u7406\u901a\u9053\u6807\u51c6\u5316\u548c\u6743\u91cd\u6807\u51c6\u5316\u8fdb\u884c\u5fae\u6279\u91cf\u8bad\u7ec3</a>\u300b\u3002\u6211\u4eec\u8fd8\u6709\u4e00\u4e2a<a href=\"../weight_standardization/index.html\">\u5e26\u6ce8\u91ca\u7684\u91cd\u91cf\u6807\u51c6\u5316\u5b9e\u73b0\u65b9\u6848</a>\u3002</p>\n<p>\u6279\u5904\u7406\u901a\u9053\u6807\u51c6\u5316\u5148\u6267\u884c\u6279\u91cf\u6807\u51c6\u5316\uff0c\u7136\u540e\u8fdb\u884c\u4fe1\u9053\u6807\u51c6\u5316\uff08\u7c7b\u4f3c\u4e8e<a href=\"../group_norm/index.html\">\u7ec4\u6807\u51c6\u5316</a>\uff09\u3002\u5f53\u6279\u6b21\u5927\u5c0f\u5f88\u5c0f\u65f6\uff0c\u4f7f\u7528\u8fd0\u884c\u5747\u503c\u548c\u65b9\u5dee\u8fdb\u884c\u6279\u91cf\u6807\u51c6\u5316\u3002</p>\n<p><a href=\"../weight_standardization/experiment.html\">\u4ee5\u4e0b\u662f\u8bad\u7ec3 VGG \u7f51\u7edc\u7684\u8bad\u7ec3\u4ee3\u7801</a>\uff0c\u8be5\u7f51\u7edc\u4f7f\u7528\u6743\u91cd\u6807\u51c6\u5316\u5bf9 CIFAR-10 \u6570\u636e\u8fdb\u884c\u5206\u7c7b\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/weight_standardization/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Batch-Channel Normalization</h2>\n<p>This first performs a batch normalization - either <a href=\"../batch_norm/index.html\">normal batch norm</a> or a batch norm with estimated mean and variance (exponential mean/variance over multiple batches). Then a channel normalization performed.</p>\n": "<h2>\u6279\u91cf\u4fe1\u9053\u89c4\u8303\u5316</h2>\n<p>\u8fd9\u9996\u5148\u6267\u884c\u6279\u6b21\u5f52\u4e00\u5316\u2014\u2014\u6b63<a href=\"../batch_norm/index.html\">\u6001\u6279\u6b21\u8303</a>\u6570\u6216\u5177\u6709\u4f30\u8ba1\u5747\u503c\u548c\u65b9\u5dee\uff08\u591a\u4e2a\u6279\u6b21\u7684\u6307\u6570\u5747\u503c/\u65b9\u5dee\uff09\u7684\u6279\u6b21\u8303\u6570\u3002\u7136\u540e\u6267\u884c\u4e86\u4fe1\u9053\u6807\u51c6\u5316\u3002</p>\n",
|
||||
"<h2>Channel Normalization</h2>\n<p>This is similar to <a href=\"../group_norm/index.html\">Group Normalization</a> but affine transform is done group wise.</p>\n": "<h2>\u9891\u9053\u89c4\u8303\u5316</h2>\n<p>\u8fd9\u4e0e<a href=\"../group_norm/index.html\">\u7ec4\u5f52\u4e00\u5316</a>\u7c7b\u4f3c\uff0c\u4f46\u4eff\u5c04\u53d8\u6362\u662f\u6309\u7ec4\u8fdb\u884c\u7684\u3002</p>\n",
|
||||
"<h2>Estimated Batch Normalization</h2>\n<p>When input <span translate=no>_^_0_^_</span> is a batch of image representations, where <span translate=no>_^_1_^_</span> is the batch size, <span translate=no>_^_2_^_</span> is the number of channels, <span translate=no>_^_3_^_</span> is the height and <span translate=no>_^_4_^_</span> is the width. <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>.</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>where,</p>\n<span translate=no>_^_8_^_</span><p>are the running mean and variances. <span translate=no>_^_9_^_</span> is the momentum for calculating the exponential mean.</p>\n": "<h2>\u9884\u8ba1\u6279\u6b21\u89c4\u8303\u5316</h2>\n<p>\u5f53\u8f93\u5165<span translate=no>_^_0_^_</span>\u662f\u4e00\u6279\u56fe\u50cf\u8868\u793a\u65f6\uff0c\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u6279\u6b21\u5927\u5c0f\uff0c<span translate=no>_^_2_^_</span>\u662f\u901a\u9053\u6570\uff0c<span translate=no>_^_3_^_</span>\u662f\u9ad8\u5ea6\u548c<span translate=no>_^_4_^_</span>\u662f\u5bbd\u5ea6\u3002<span translate=no>_^_5_^_</span>\u548c<span translate=no>_^_6_^_</span>\u3002</p>\n<p><span translate=no>_^_7_^_</span></p>\n<p>\u5728\u54ea\u91cc\uff0c</p>\n<span translate=no>_^_8_^_</span><p>\u662f\u8fd0\u884c\u5747\u503c\u548c\u65b9\u5dee\u3002<span translate=no>_^_9_^_</span>\u662f\u8ba1\u7b97\u6307\u6570\u5747\u503c\u7684\u52a8\u91cf\u3002</p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u5f20\u91cf<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u8868\u793a\u4efb\u610f\u6570\u91cf\uff08\u53ef\u80fd\u4e3a 0\uff09\u7684\u7ef4\u5ea6\u3002\u4f8b\u5982\uff0c\u5728\u56fe\u50cf\uff082D\uff09\u5377\u79ef\u4e2d\uff0c\u8fd9\u5c06\u662f<span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p>Calculate the mean across first and last dimensions; <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u7b2c\u4e00\u7ef4\u548c\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684\u5e73\u5747\u503c\uff1b<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the mean across last dimension; i.e. the means for each sample and channel group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684\u5747\u503c\uff1b\u5373\u6bcf\u4e2a\u6837\u672c\u548c\u901a\u9053\u7ec4\u7684\u5747\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimensions; <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u7b2c\u4e00\u7ef4\u548c\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684\u5747\u65b9\u503c\uff1b<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean across last dimension; i.e. the means for each sample and channel group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684\u5747\u65b9\u503c\uff1b\u5373\u6bcf\u4e2a\u6837\u672c\u548c\u901a\u9053\u7ec4\u7684\u5747\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Channel normalization </p>\n": "<p>\u4fe1\u9053\u89c4\u8303\u5316</p>\n",
|
||||
"<p>Channel wise transformation parameters </p>\n": "<p>\u9891\u9053\u53d8\u6362\u53c2\u6570</p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u83b7\u53d6\u6279\u6b21\u5927\u5c0f</p>\n",
|
||||
"<p>Keep old shape </p>\n": "<p>\u4fdd\u6301\u65e7\u7684\u5f62\u72b6</p>\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u4fdd\u6301\u539f\u59cb\u5f62\u72b6</p>\n",
|
||||
"<p>No backpropagation through <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u6ca1\u6709\u901a\u8fc7<span translate=no>_^_0_^_</span>\u548c\u7684\u53cd\u5411\u4f20\u64ad<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u89c4\u8303\u5316<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Parameters for affine transformation.</p>\n<p><em>Note that these transforms are per group, unlike in group norm where they are transformed channel-wise.</em> </p>\n": "<p>\u4eff\u5c04\u53d8\u6362\u7684\u53c2\u6570\u3002</p>\n<p><em>\u8bf7\u6ce8\u610f\uff0c\u8fd9\u4e9b\u53d8\u6362\u662f\u6309\u7ec4\u8fdb\u884c\u7684\uff0c\u8fd9\u4e0e\u7ec4\u89c4\u8303\u4e0d\u540c\uff0c\u5b83\u4eec\u662f\u6309\u901a\u9053\u53d8\u6362\u7684\u3002</em></p>\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u91cd\u5851\u6210<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u91cd\u5851\u4e3a\u539f\u59cb\u5f62\u72b6\u7136\u540e\u8fd4\u56de</p>\n",
|
||||
"<p>Sanity check to make sure the number of features is correct </p>\n": "<p>\u8fdb\u884c\u5065\u5168\u6027\u68c0\u67e5\u4ee5\u786e\u4fdd\u8981\u7d20\u6570\u91cf\u6b63\u786e</p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u8fdb\u884c\u5065\u5168\u6027\u68c0\u67e5\u4ee5\u786e\u4fdd\u8981\u7d20\u6570\u91cf\u76f8\u540c</p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7f29\u653e\u548c\u79fb\u52a8<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Scale and shift group-wise <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6309\u7ec4\u7f29\u653e\u548c\u79fb\u52a8<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Tensors for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u548c\u7684\u5f20\u91cf<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Update <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in training mode only </p>\n": "<p>\u66f4\u65b0<span translate=no>_^_0_^_</span>\u4e14\u4ec5<span translate=no>_^_1_^_</span>\u5728\u8bad\u7ec3\u6a21\u5f0f\u4e0b</p>\n",
|
||||
"<p>Update exponential moving averages</p>\n<span translate=no>_^_0_^_</span><p> </p>\n": "<p>\u66f4\u65b0\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf</p>\n<span translate=no>_^_0_^_</span><p></p>\n",
|
||||
"<p>Use estimated batch norm or normal batch norm. </p>\n": "<p>\u4f7f\u7528\u4f30\u8ba1\u7684\u6279\u6b21\u89c4\u8303\u6216\u666e\u901a\u6279\u6b21\u89c4\u8303\u3002</p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6bcf\u4e2a\u8981\u7d20\u7684\u65b9\u5dee<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Variance for each sample and feature group <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6bcf\u4e2a\u6837\u672c\u548c\u7279\u5f81\u7ec4\u7684\u65b9\u5dee<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_5_^_</span> is whether to use running mean and variance for batch norm</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8f93\u5165\u4e2d\u7684\u8981\u7d20\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f<span translate=no>_^_2_^_</span>\uff0c<span translate=no>_^_3_^_</span>\u7528\u4e8e\u6570\u503c\u7a33\u5b9a\u6027</li>\n<li><span translate=no>_^_4_^_</span>\u662f\u53d6\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u7684\u52a8\u91cf</li>\n<li><span translate=no>_^_5_^_</span>\u662f\u5426\u4f7f\u7528\u8fd0\u884c\u5747\u503c\u548c\u65b9\u5dee\u4f5c\u4e3a\u6279\u6b21\u8303\u6570</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is the number of groups the features are divided into </li>\n<li><span translate=no>_^_2_^_</span> is <span translate=no>_^_3_^_</span>, used in <span translate=no>_^_4_^_</span> for numerical stability </li>\n<li><span translate=no>_^_5_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_6_^_</span> is whether to use running mean and variance for batch norm</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8f93\u5165\u4e2d\u7684\u8981\u7d20\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u8981\u7d20\u88ab\u5212\u5206\u5230\u7684\u7ec4\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_2_^_</span>\u662f<span translate=no>_^_3_^_</span>\uff0c<span translate=no>_^_4_^_</span>\u7528\u4e8e\u6570\u503c\u7a33\u5b9a\u6027</li>\n<li><span translate=no>_^_5_^_</span>\u662f\u53d6\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u7684\u52a8\u91cf</li>\n<li><span translate=no>_^_6_^_</span>\u662f\u5426\u4f7f\u7528\u8fd0\u884c\u5747\u503c\u548c\u65b9\u5dee\u4f5c\u4e3a\u6279\u6b21\u8303\u6570</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of groups the features are divided into </li>\n<li><span translate=no>_^_1_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_2_^_</span> is <span translate=no>_^_3_^_</span>, used in <span translate=no>_^_4_^_</span> for numerical stability </li>\n<li><span translate=no>_^_5_^_</span> is whether to scale and shift the normalized value</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8981\u7d20\u88ab\u5212\u5206\u5230\u7684\u7ec4\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u8f93\u5165\u4e2d\u7684\u8981\u7d20\u6570</li>\n<li><span translate=no>_^_2_^_</span>\u662f<span translate=no>_^_3_^_</span>\uff0c<span translate=no>_^_4_^_</span>\u7528\u4e8e\u6570\u503c\u7a33\u5b9a\u6027</li>\n<li><span translate=no>_^_5_^_</span>\u662f\u5426\u7f29\u653e\u548c\u79fb\u52a8\u5f52\u4e00\u5316\u503c</li></ul>\n",
|
||||
"A PyTorch implementation/tutorial of Batch-Channel Normalization.": "\u6279\u91cf\u4fe1\u9053\u89c4\u8303\u5316\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002",
|
||||
"Batch-Channel Normalization": "\u6279\u91cf\u4fe1\u9053\u89c4\u8303\u5316"
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"<h1>Batch Normalization</h1>\n": "<h1>\u30d0\u30c3\u30c1\u6b63\u898f\u5316</h1>\n",
|
||||
"<h2>Batch Normalization Layer</h2>\n<p>Batch normalization layer <span translate=no>_^_0_^_</span> normalizes the input <span translate=no>_^_1_^_</span> as follows:</p>\n<p>When input <span translate=no>_^_2_^_</span> is a batch of image representations, where <span translate=no>_^_3_^_</span> is the batch size, <span translate=no>_^_4_^_</span> is the number of channels, <span translate=no>_^_5_^_</span> is the height and <span translate=no>_^_6_^_</span> is the width. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span></p>\n<p>When input <span translate=no>_^_10_^_</span> is a batch of embeddings, where <span translate=no>_^_11_^_</span> is the batch size and <span translate=no>_^_12_^_</span> is the number of features. <span translate=no>_^_13_^_</span> and <span translate=no>_^_14_^_</span>. <span translate=no>_^_15_^_</span></p>\n<p>When input <span translate=no>_^_16_^_</span> is a batch of a sequence embeddings, where <span translate=no>_^_17_^_</span> is the batch size, <span translate=no>_^_18_^_</span> is the number of features, and <span translate=no>_^_19_^_</span> is the length of the sequence. <span translate=no>_^_20_^_</span> and <span translate=no>_^_21_^_</span>. <span translate=no>_^_22_^_</span></p>\n": "<h2>\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc</h2>\n<p>\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u5c64\u306f\u3001<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u6b21\u306e\u3088\u3046\u306b\u5165\u529b\u3092\u6b63\u898f\u5316\u3057\u307e\u3059\u3002</p>\n<p><span translate=no>_^_2_^_</span>\u5165\u529b\u304c\u30a4\u30e1\u30fc\u30b8\u8868\u73fe\u306e\u30d0\u30c3\u30c1\u306e\u5834\u5408\u3001<span translate=no>_^_3_^_</span>\u306f\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3001<span translate=no>_^_4_^_</span>\u306f\u30c1\u30e3\u30cd\u30eb\u6570\u3001<span translate=no>_^_5_^_</span>\u306f\u9ad8\u3055\u3001<span translate=no>_^_6_^_</span>\u306f\u5e45\u3067\u3059\u3002<span translate=no>_^_7_^_</span>\u3068<span translate=no>_^_8_^_</span>\u3002<span translate=no>_^_9_^_</span></p>\n<p>\u5165\u529b\u304c\u57cb\u3081\u8fbc\u307f\u306e\u30d0\u30c3\u30c1\u306e\u5834\u5408\u3001<span translate=no>_^_10_^_</span>\u306f\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3001<span translate=no>_^_11_^_</span><span translate=no>_^_12_^_</span>\u306f\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u6570\u3067\u3059\u3002<span translate=no>_^_13_^_</span>\u3068<span translate=no>_^_14_^_</span>\u3002<span translate=no>_^_15_^_</span></p>\n<p><span translate=no>_^_16_^_</span>\u5165\u529b\u304c\u30b7\u30fc\u30b1\u30f3\u30b9\u57cb\u3081\u8fbc\u307f\u306e\u30d0\u30c3\u30c1\u306e\u5834\u5408\u3001<span translate=no>_^_17_^_</span>\u306f\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3001<span translate=no>_^_18_^_</span>\u306f\u30d5\u30a3\u30fc\u30c1\u30e3\u6570\u3001<span translate=no>_^_19_^_</span>\u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3067\u3059\u3002<span translate=no>_^_20_^_</span>\u3068<span translate=no>_^_21_^_</span>\u3002<span translate=no>_^_22_^_</span></p>\n",
|
||||
"<h2>Inference</h2>\n": "<h2>\u63a8\u8ad6</h2>\n",
|
||||
"<h2>Normalization</h2>\n": "<h2>\u30ce\u30fc\u30de\u30e9\u30a4\u30bc\u30fc\u30b7\u30e7\u30f3</h2>\n",
|
||||
"<h3>Batch Normalization</h3>\n": "<h3>\u30d0\u30c3\u30c1\u6b63\u898f\u5316</h3>\n",
|
||||
"<h3>Internal Covariate Shift</h3>\n": "<h3>\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8</h3>\n",
|
||||
"<h3>Normalizing outside gradient computation doesn't work</h3>\n": "<h3>\u5916\u90e8\u52fe\u914d\u8a08\u7b97\u306e\u6b63\u898f\u5316\u306f\u6a5f\u80fd\u3057\u307e\u305b\u3093</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5f62\u72b6\u306e\u30c6\u30f3\u30bd\u30eb\u3067\u3059\u3002<span translate=no>_^_2_^_</span>\u4efb\u610f\u306e\u6570 (0 \u306e\u5834\u5408\u3082\u3042\u308a\u307e\u3059) \u306e\u6b21\u5143\u3092\u793a\u3057\u307e\u3059\u3002\u305f\u3068\u3048\u3070\u3001\u753b\u50cf (2D) \u306e\u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u3067\u306f\u3001\u6b21\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059</p>\u3002<span translate=no>_^_3_^_</span>\n",
|
||||
"<p> Simple test</p>\n": "<p>\u7c21\u5358\u306a\u30c6\u30b9\u30c8</p>\n",
|
||||
"<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<p>Batch normalization also makes the back propagation invariant to the scale of the weights and empirically it improves generalization, so it has regularization effects too.</p>\n": "<p>\u307e\u305f\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3067\u306f\u9006\u4f1d\u64ad\u304c\u91cd\u307f\u306e\u30b9\u30b1\u30fc\u30eb\u306b\u5bfe\u3057\u3066\u4e0d\u5909\u306b\u306a\u308a\u3001\u7d4c\u9a13\u7684\u306b\u30b8\u30a7\u30cd\u30e9\u30e9\u30a4\u30ba\u304c\u6539\u5584\u3055\u308c\u308b\u305f\u3081\u3001\u6b63\u5247\u5316\u52b9\u679c\u3082\u3042\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<p>By stabilizing the distribution, batch normalization minimizes the internal covariate shift.</p>\n": "<p>\u5206\u5e03\u3092\u5b89\u5b9a\u3055\u305b\u308b\u3053\u3068\u306b\u3088\u308a\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306f\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u3092\u6700\u5c0f\u9650\u306b\u6291\u3048\u307e\u3059\u3002</p>\n",
|
||||
"<p>Calculate the mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u521d\u306e\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u3068\u6700\u5f8c\u306e\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3\u306e\u5e73\u5747\u3001\u3064\u307e\u308a\u5404\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u521d\u3068\u6700\u5f8c\u306e\u6b21\u5143\u306e\u4e8c\u4e57\u5e73\u5747\u3001\u3064\u307e\u308a\u5404\u7279\u5fb4\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create buffers to store exponential moving averages of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5e73\u5747\u3068\u5206\u6563\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3092\u683c\u7d0d\u3059\u308b\u30d0\u30c3\u30d5\u30a1\u30fc\u306e\u4f5c\u6210 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Create parameters for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for scale and shift </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u30b9\u30b1\u30fc\u30eb\u3068\u30b7\u30d5\u30c8\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u3068\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Here's <a href=\"mnist.html\">the training code</a> and a notebook for training a CNN classifier that uses batch normalization for MNIST dataset.</p>\n": "<p>\u4ee5\u4e0b\u306f<a href=\"mnist.html\">\u3001MNIST \u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3059\u308b CNN \u5206\u985e\u5668\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3068\u30ce\u30fc\u30c8\u30d6\u30c3\u30af\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>Internal covariate shift will adversely affect training speed because the later layers (<span translate=no>_^_0_^_</span> in the above example) have to adapt to this shifted distribution.</p>\n": "<p>\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u306f\u3001\u5f8c\u306e\u5c64\uff08<span translate=no>_^_0_^_</span>\u4e0a\u306e\u4f8b\uff09\u304c\u3053\u306e\u30b7\u30d5\u30c8\u3057\u305f\u5206\u5e03\u306b\u9069\u5fdc\u3057\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u305f\u3081\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u901f\u5ea6\u306b\u60aa\u5f71\u97ff\u3092\u53ca\u307c\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>It is known that whitening improves training speed and convergence. <em>Whitening</em> is linearly transforming inputs to have zero mean, unit variance, and be uncorrelated.</p>\n": "<p>\u30db\u30ef\u30a4\u30c8\u30cb\u30f3\u30b0\u306f\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u30b9\u30d4\u30fc\u30c9\u3068\u30b3\u30f3\u30d0\u30fc\u30b8\u30a7\u30f3\u30b9\u3092\u5411\u4e0a\u3055\u305b\u308b\u3053\u3068\u304c\u77e5\u3089\u308c\u3066\u3044\u307e\u3059\u3002<em>\u30db\u30ef\u30a4\u30c8\u30cb\u30f3\u30b0\u3068\u306f</em>\u3001\u5165\u529b\u3092\u5e73\u5747\u304c\u30bc\u30ed\u3001\u5358\u4f4d\u5206\u6563\u3001\u7121\u76f8\u95a2\u306b\u306a\u308b\u3088\u3046\u306b\u7dda\u5f62\u306b\u5909\u63db\u3059\u308b\u3053\u3068\u3067\u3059</p>\u3002\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u5143\u306e\u5f62\u3092\u4fdd\u3064</p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30ce\u30fc\u30de\u30e9\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Normalizing each feature to zero mean and unit variance could affect what the layer can represent. As an example paper illustrates that, if the inputs to a sigmoid are normalized most of it will be within <span translate=no>_^_0_^_</span> range where the sigmoid is linear. To overcome this each feature is scaled and shifted by two trained parameters <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span>. <span translate=no>_^_3_^_</span> where <span translate=no>_^_4_^_</span> is the output of the batch normalization layer.</p>\n": "<p>\u5404\u7279\u5fb4\u91cf\u3092\u5e73\u5747\u30bc\u30ed\u3068\u5358\u4f4d\u5206\u6563\u306b\u6b63\u898f\u5316\u3059\u308b\u3068\u3001\u30ec\u30a4\u30e4\u30fc\u304c\u8868\u73fe\u3067\u304d\u308b\u5185\u5bb9\u306b\u5f71\u97ff\u3059\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\u3002\u4f8b\u793a\u3057\u3066\u3044\u308b\u3088\u3046\u306b\u3001\u30b7\u30b0\u30e2\u30a4\u30c9\u3078\u306e\u5165\u529b\u304c\u6b63\u898f\u5316\u3055\u308c\u308b\u3068\u3001<span translate=no>_^_0_^_</span>\u305d\u306e\u307b\u3068\u3093\u3069\u306f\u30b7\u30b0\u30e2\u30a4\u30c9\u304c\u7dda\u5f62\u3067\u3042\u308b\u7bc4\u56f2\u5185\u306b\u306a\u308a\u307e\u3059\u3002\u3053\u308c\u3092\u89e3\u6c7a\u3059\u308b\u305f\u3081\u306b\u3001\u5404\u6a5f\u80fd\u306e\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3068\u30b7\u30d5\u30c8\u3092\u5b66\u7fd2\u6e08\u307f\u306e 2 <span translate=no>_^_1_^_</span> \u3064\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3068\u3067\u8abf\u6574\u3057\u307e\u3059\u3002<span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u3053\u3053\u3067<span translate=no>_^_4_^_</span>\u3001\u306f\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u5c64\u306e\u51fa\u529b\u3067\u3059</p>\u3002\n",
|
||||
"<p>Normalizing outside the gradient computation using pre-computed (detached) means and variances doesn't work. For instance. (ignoring variance), let <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> is a trained bias and <span translate=no>_^_3_^_</span> is an outside gradient computation (pre-computed constant).</p>\n": "<p>\u4e8b\u524d\u306b\u8a08\u7b97\u3055\u308c\u305f\uff08\u5206\u96e2\u3055\u308c\u305f\uff09\u5e73\u5747\u3068\u5206\u6563\u3092\u4f7f\u7528\u3057\u3066\u52fe\u914d\u8a08\u7b97\u306e\u5916\u3067\u6b63\u898f\u5316\u3059\u308b\u3053\u3068\u306f\u3067\u304d\u307e\u305b\u3093\u3002\u4f8b\u3048\u3070\u3002(\u5206\u6563\u306f\u7121\u8996)\u3001<span translate=no>_^_0_^_</span>\u3053\u3053\u3067\u3001<span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> \u306f\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u6e08\u307f\u306e\u30d0\u30a4\u30a2\u30b9\u3067\u3001\u5916\u90e8\u52fe\u914d\u8a08\u7b97 (\u4e8b\u524d\u306b\u8a08\u7b97\u3055\u308c\u305f\u5b9a\u6570) <span translate=no>_^_3_^_</span> \u3067\u3059</p>\u3002\n",
|
||||
"<p>Note that <span translate=no>_^_0_^_</span> has no effect on <span translate=no>_^_1_^_</span>. Therefore, <span translate=no>_^_2_^_</span> will increase or decrease based <span translate=no>_^_3_^_</span>, and keep on growing indefinitely in each training update. The paper notes that similar explosions happen with variances.</p>\n": "<p><span translate=no>_^_0_^_</span>\u306b\u306f\u5f71\u97ff\u3057\u306a\u3044\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044<span translate=no>_^_1_^_</span>\u3002\u3057\u305f\u304c\u3063\u3066\u3001<span translate=no>_^_2_^_</span>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u66f4\u65b0\u3059\u308b\u305f\u3073\u306b\u5897\u52a0\u307e\u305f\u306f\u6e1b\u5c11\u3057<span translate=no>_^_3_^_</span>\u3001\u7121\u671f\u9650\u306b\u6210\u9577\u3057\u7d9a\u3051\u307e\u3059\u3002\u3053\u306e\u8ad6\u6587\u306f\u3001\u540c\u69d8\u306e\u7206\u767a\u306b\u306f\u3070\u3089\u3064\u304d\u304c\u3042\u308b\u3068\u8ff0\u3079\u3066\u3044\u307e\u3059</p>\u3002\n",
|
||||
"<p>Note that when applying batch normalization after a linear transform like <span translate=no>_^_0_^_</span> the bias parameter <span translate=no>_^_1_^_</span> gets cancelled due to normalization. So you can and should omit bias parameter in linear transforms right before the batch normalization.</p>\n": "<p><span translate=no>_^_0_^_</span>\u7dda\u5f62\u5909\u63db\u306e\u3088\u3046\u306a\u7dda\u5f62\u5909\u63db\u306e\u5f8c\u306b\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092\u9069\u7528\u3059\u308b\u3068\u3001<span translate=no>_^_1_^_</span>\u6b63\u898f\u5316\u306b\u3088\u308a\u30d0\u30a4\u30a2\u30b9\u30d1\u30e9\u30e1\u30fc\u30bf\u304c\u30ad\u30e3\u30f3\u30bb\u30eb\u3055\u308c\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002\u305d\u306e\u305f\u3081\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306e\u76f4\u524d\u306b\u7dda\u5f62\u5909\u63db\u306e\u30d0\u30a4\u30a2\u30b9\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u7701\u7565\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u3001\u307e\u305f\u7701\u7565\u3059\u3079\u304d\u3067\u3059</p>\u3002\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5f62\u3092\u5909\u3048\u3066 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u5143\u306e\u5f62\u306b\u623b\u3057\u3066\u623b\u3059</p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u6a5f\u80fd\u306e\u6570\u304c\u540c\u3058\u3067\u3042\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3059\u308b\u305f\u3081\u306e\u30b5\u30cb\u30c6\u30a3\u30c1\u30a7\u30c3\u30af</p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30b1\u30fc\u30eb\u3068\u30b7\u30d5\u30c8 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The paper defines <em>Internal Covariate Shift</em> as the change in the distribution of network activations due to the change in network parameters during training. For example, let's say there are two layers <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>. During the beginning of the training <span translate=no>_^_2_^_</span> outputs (inputs to <span translate=no>_^_3_^_</span>) could be in distribution <span translate=no>_^_4_^_</span>. Then, after some training steps, it could move to <span translate=no>_^_5_^_</span>. This is <em>internal covariate shift</em>.</p>\n": "<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<em>\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u3092</em>\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306e\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u306e\u5909\u5316\u306b\u3088\u308b\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306e\u5206\u5e03\u306e\u5909\u5316\u3068\u3057\u3066\u5b9a\u7fa9\u3057\u3066\u3044\u307e\u3059\u3002\u305f\u3068\u3048\u3070\u3001<span translate=no>_^_0_^_</span>\u3068\u306e 2 \u3064\u306e\u30ec\u30a4\u30e4\u30fc\u304c\u3042\u308b\u3068\u3057\u307e\u3059<span translate=no>_^_1_^_</span>\u3002\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u958b\u59cb\u6642\u306b\u3001<span translate=no>_^_2_^_</span>\u30a2\u30a6\u30c8\u30d7\u30c3\u30c8\uff08\u3078\u306e\u30a4\u30f3\u30d7\u30c3\u30c8<span translate=no>_^_3_^_</span>\uff09\u304c\u914d\u5e03\u3055\u308c\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059<span translate=no>_^_4_^_</span>\u3002\u305d\u306e\u5f8c\u3001\u3044\u304f\u3064\u304b\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u624b\u9806\u3092\u5b9f\u884c\u3059\u308b\u3068\u3001\u306b\u79fb\u52d5\u3059\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059<span translate=no>_^_5_^_</span>\u3002<em>\u3053\u308c\u306f\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u3067\u3059</em></p>\u3002\n",
|
||||
"<p>The paper introduces a simplified version which they call <em>Batch Normalization</em>. First simplification is that it normalizes each feature independently to have zero mean and unit variance: <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span>-dimensional input.</p>\n": "<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<em>\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3068\u547c\u3070\u308c\u308b\u7c21\u7565\u7248\u3092\u7d39\u4ecb\u3057\u3066\u3044\u307e\u3059</em>\u30021 \u3064\u76ee\u306e\u7c21\u7565\u5316\u306f\u3001\u5404\u7279\u5fb4\u91cf\u3092\u72ec\u7acb\u3057\u3066\u5e73\u5747\u304c 0\u3001\u5358\u4f4d\u5206\u6563\u306b\u306a\u308b\u3088\u3046\u306b\u6b63\u898f\u5316\u3059\u308b\u3053\u3068\u3067\u3059\u3002<span translate=no>_^_0_^_</span>\u3053\u3053\u3067\u3001\u306f <span translate=no>_^_1_^_</span>-\u6b21\u5143\u306e\u5165\u529b\u3067\u3059</p>\u3002<span translate=no>_^_2_^_</span>\n",
|
||||
"<p>The second simplification is to use estimates of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> from the mini-batch for normalization; instead of calculating the mean and variance across the whole dataset.</p>\n": "<p>2 \u3064\u76ee\u306e\u7c21\u7565\u5316\u306f\u3001<span translate=no>_^_0_^_</span>\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u5168\u4f53\u306e\u5e73\u5747\u3068\u5206\u6563\u3092\u8a08\u7b97\u3059\u308b\u306e\u3067\u306f\u306a\u304f\u3001<span translate=no>_^_1_^_</span>\u30df\u30cb\u30d0\u30c3\u30c1\u304b\u3089\u306e\u5e73\u5747\u3068\u5206\u6563\u306e\u63a8\u5b9a\u5024\u3092\u6b63\u898f\u5316\u306b\u4f7f\u7528\u3059\u308b\u3053\u3068\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch Normalization from paper <a href=\"https://arxiv.org/abs/1502.03167\">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u300c\u30d0\u30c3\u30c1\u6b63\u898f\u5316<a href=\"https://arxiv.org/abs/1502.03167\">:\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u3092\u6e1b\u3089\u3059\u3053\u3068\u306b\u3088\u308b\u30c7\u30a3\u30fc\u30d7\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u9ad8\u901f\u5316\u300d<a href=\"https://pytorch.org\">\u3068\u3044\u3046\u8ad6\u6587\u304b\u3089\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n",
|
||||
"<p>Update exponential moving averages </p>\n": "<p>\u6307\u6570\u79fb\u52d5\u5e73\u5747\u306e\u66f4\u65b0</p>\n",
|
||||
"<p>Use exponential moving averages as estimates </p>\n": "<p>\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3092\u63a8\u5b9a\u5024\u3068\u3057\u3066\u4f7f\u7528</p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5404\u6a5f\u80fd\u306e\u5dee\u7570 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>We need to know <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in order to perform the normalization. So during inference, you either need to go through the whole (or part of) dataset and find the mean and variance, or you can use an estimate calculated during training. The usual practice is to calculate an exponential moving average of mean and variance during the training phase and use that for inference.</p>\n": "<p>\u6b63\u898f\u5316\u3092\u5b9f\u884c\u3059\u308b\u306b\u306f<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3001\u3068\u3092\u77e5\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\u305d\u306e\u305f\u3081\u3001\u63a8\u8ad6\u6642\u306b\u306f\u3001\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u5168\u4f53 (\u307e\u305f\u306f\u4e00\u90e8) \u3092\u8abf\u3079\u3066\u5e73\u5747\u3068\u5206\u6563\u3092\u6c42\u3081\u308b\u304b\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\u8a08\u7b97\u3055\u308c\u305f\u63a8\u5b9a\u5024\u3092\u4f7f\u7528\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\u901a\u5e38\u306f\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u6bb5\u968e\u3067\u5e73\u5747\u3068\u5206\u6563\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3092\u8a08\u7b97\u3057\u3001\u305d\u308c\u3092\u63a8\u8ad6\u306b\u4f7f\u7528\u3057\u307e\u3059</p>\u3002\n",
|
||||
"<p>We will calculate the mini-batch mean and variance if we are in training mode or if we have not tracked exponential moving averages </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30e2\u30fc\u30c9\u306e\u5834\u5408\u3001\u307e\u305f\u306f\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3092\u8ffd\u8de1\u3057\u3066\u3044\u306a\u3044\u5834\u5408\u306f\u3001\u30df\u30cb\u30d0\u30c3\u30c1\u306e\u5e73\u5747\u3068\u5206\u6563\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Whitening is computationally expensive because you need to de-correlate and the gradients must flow through the full whitening calculation.</p>\n": "<p>\u30db\u30ef\u30a4\u30c8\u30cb\u30f3\u30b0\u306f\u3001\u76f8\u95a2\u3092\u306a\u304f\u3059\u5fc5\u8981\u304c\u3042\u308a\u3001\u52fe\u914d\u304c\u30db\u30ef\u30a4\u30c8\u30cb\u30f3\u30b0\u306e\u8a08\u7b97\u5168\u4f53\u3092\u901a\u308b\u5fc5\u8981\u304c\u3042\u308b\u305f\u3081\u3001\u8a08\u7b97\u91cf\u304c\u591a\u304f\u306a\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_5_^_</span> is whether to scale and shift the normalized value </li>\n<li><span translate=no>_^_6_^_</span> is whether to calculate the moving averages or mean and variance</li></ul>\n<p>We've tried to use the same names for arguments as PyTorch <span translate=no>_^_7_^_</span> implementation.</p>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u5165\u529b\u5185\u306e\u7279\u5fb4\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span><span translate=no>_^_3_^_</span>\u6570\u5024\u306e\u5b89\u5b9a\u6027\u306e\u305f\u3081\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3092\u53d6\u308b\u3068\u304d\u306e\u52e2\u3044\u3067\u3059</li>\n<li><span translate=no>_^_5_^_</span>\u6b63\u898f\u5316\u3055\u308c\u305f\u5024\u3092\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3057\u3066\u30b7\u30d5\u30c8\u3059\u308b\u304b\u3069\u3046\u304b\u3067\u3059</li>\n<li><span translate=no>_^_6_^_</span>\u79fb\u52d5\u5e73\u5747\u3092\u8a08\u7b97\u3059\u308b\u304b\u3001\u5e73\u5747\u3068\u5206\u6563\u3092\u8a08\u7b97\u3059\u308b\u304b\u3067\u3059</li></ul>\n<p>\u5f15\u6570\u306b\u306f PyTorch <span translate=no>_^_7_^_</span> \u5b9f\u88c5\u3068\u540c\u3058\u540d\u524d\u3092\u4f7f\u7528\u3057\u3088\u3046\u3068\u3057\u307e\u3057\u305f\u3002</p>\n",
|
||||
"A PyTorch implementation/tutorial of batch normalization.": "\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306e PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"Batch Normalization": "\u30d0\u30c3\u30c1\u6b63\u898f\u5316"
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"<h1>Batch Normalization</h1>\n": "<h1>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</h1>\n",
|
||||
"<h2>Batch Normalization Layer</h2>\n<p>Batch normalization layer <span translate=no>_^_0_^_</span> normalizes the input <span translate=no>_^_1_^_</span> as follows:</p>\n<p>When input <span translate=no>_^_2_^_</span> is a batch of image representations, where <span translate=no>_^_3_^_</span> is the batch size, <span translate=no>_^_4_^_</span> is the number of channels, <span translate=no>_^_5_^_</span> is the height and <span translate=no>_^_6_^_</span> is the width. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span></p>\n<p>When input <span translate=no>_^_10_^_</span> is a batch of embeddings, where <span translate=no>_^_11_^_</span> is the batch size and <span translate=no>_^_12_^_</span> is the number of features. <span translate=no>_^_13_^_</span> and <span translate=no>_^_14_^_</span>. <span translate=no>_^_15_^_</span></p>\n<p>When input <span translate=no>_^_16_^_</span> is a batch of a sequence embeddings, where <span translate=no>_^_17_^_</span> is the batch size, <span translate=no>_^_18_^_</span> is the number of features, and <span translate=no>_^_19_^_</span> is the length of the sequence. <span translate=no>_^_20_^_</span> and <span translate=no>_^_21_^_</span>. <span translate=no>_^_22_^_</span></p>\n": "<h2>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba</h2>\n<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0d86\u0daf\u0dcf\u0db1\u0dba \u0db4\u0dc4\u0dad <span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dd2\u0daf\u0dd2 <span translate=no>_^_0_^_</span> \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0dba\u0dd2:</p>\n<p><span translate=no>_^_2_^_</span> \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dbb\u0dd6\u0db4 \u0db1\u0dd2\u0dbb\u0dd6\u0db4\u0dab \u0dc3\u0db8\u0dd6\u0dc4\u0dba\u0d9a\u0dca <span translate=no>_^_3_^_</span> \u0dc0\u0db1 \u0dc0\u0dd2\u0da7, \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf, <span translate=no>_^_4_^_</span> \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0d9c\u0dab\u0db1, <span translate=no>_^_5_^_</span> \u0d8b\u0dc3 \u0dc3\u0dc4 <span translate=no>_^_6_^_</span> \u0db4\u0dc5\u0dbd \u0dc0\u0dda. <span translate=no>_^_7_^_</span> \u0dc3\u0dc4 <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span></p>\n<p><span translate=no>_^_10_^_</span> \u0d86\u0daf\u0dcf\u0db1\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0db8\u0dd6\u0dc4\u0dba\u0d9a\u0dca <span translate=no>_^_11_^_</span> \u0dc0\u0db1 \u0dc0\u0dd2\u0da7, \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf \u0dc3\u0dc4 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 <span translate=no>_^_12_^_</span> \u0dc0\u0dda. <span translate=no>_^_13_^_</span> \u0dc3\u0dc4 <span translate=no>_^_14_^_</span>. <span translate=no>_^_15_^_</span></p>\n<p>\u0d86\u0daf\u0dcf\u0db1\u0dba <span translate=no>_^_16_^_</span> \u0dba\u0db1\u0dd4 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0d9a\u0dcf\u0dab\u0dca\u0da9\u0dba\u0d9a\u0dca <span translate=no>_^_17_^_</span> \u0dc0\u0db1 \u0dc0\u0dd2\u0da7, \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf, \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 <span translate=no>_^_19_^_</span> \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0daf\u0dd2\u0d9c \u0dc0\u0dda <span translate=no>_^_18_^_</span> \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba. <span translate=no>_^_20_^_</span> \u0dc3\u0dc4 <span translate=no>_^_21_^_</span>. <span translate=no>_^_22_^_</span></p>\n",
|
||||
"<h2>Inference</h2>\n": "<h2>\u0d85\u0db1\u0dd4\u0db8\u0dcf\u0db1\u0dba</h2>\n",
|
||||
"<h2>Normalization</h2>\n": "<h2>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u200d\u0dba\u0d9a\u0dbb\u0dab\u0dba</h2>\n",
|
||||
"<h3>Batch Normalization</h3>\n": "<h3>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</h3>\n",
|
||||
"<h3>Internal Covariate Shift</h3>\n": "<h3>\u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb\u0d9a\u0ddd\u0dc0\u0dbb\u0dd2\u0dba\u0db1\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0</h3>\n",
|
||||
"<h3>Normalizing outside gradient computation doesn't work</h3>\n": "<h3>\u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dd2\u0da7\u0dad \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0dba\u0dd2</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p> <span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d86\u0dad\u0db1\u0dca\u0dba <span translate=no>_^_1_^_</span>\u0dc0\u0dda. <span translate=no>_^_2_^_</span> \u0d95\u0db1\u0dd1\u0db8 \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0\u0d9a\u0dca (\u0dc3\u0db8\u0dc4\u0dbb\u0dc0\u0dd2\u0da7 0) \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0daf\u0d9a\u0dca\u0dc0\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3, \u0dbb\u0dd6\u0db4\u0dba\u0d9a\u0dca (2D) \u0dc3\u0d82\u0d9a\u0ddd\u0da0\u0db1\u0dba \u0dad\u0dd4\u0dc5 \u0db8\u0dd9\u0dba \u0dc0\u0db1\u0dd4 \u0d87\u0dad <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p> Simple test</p>\n": "<p> \u0dc3\u0dbb\u0dbd\u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba</p>\n",
|
||||
"<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/011254fe647011ebbb8e0242ac1c0002\"><span translate=no>_^_1_^_</span></a></p>\n": "<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/011254fe647011ebbb8e0242ac1c0002\"> <span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<p>Batch normalization also makes the back propagation invariant to the scale of the weights and empirically it improves generalization, so it has regularization effects too.</p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db8\u0d9f\u0dd2\u0db1\u0dca \u0db4\u0dc3\u0dd4\u0db4\u0dc3 \u0db4\u0dca\u0dbb\u0da0\u0dcf\u0dbb\u0dab\u0dba \u0db6\u0dbb \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0da7 \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0d86\u0db1\u0dd4\u0db7\u0dc0\u0dd2\u0d9a \u0dbd\u0dd9\u0dc3 \u0d91\u0dba \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2, \u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d91\u0dba \u0dc0\u0dd2\u0db0\u0dd2\u0db8\u0dad\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db6\u0dbd\u0db4\u0dd1\u0db8\u0dca \u0daf \u0d87\u0dad. </p>\n",
|
||||
"<p>By stabilizing the distribution, batch normalization minimizes the internal covariate shift.</p>\n": "<p>\u0db6\u0dd9\u0daf\u0dcf\u0dc4\u0dd0\u0dbb\u0dd3\u0db8 \u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca, \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb \u0d9a\u0ddd\u0dc0\u0dd2\u0dbb\u0dda\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 \u0d85\u0dc0\u0db8 \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>Calculate the mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0dc4\u0dcf \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db8\u0dcf\u0db1\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1; \u0d91\u0db1\u0db8\u0dca \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0dbd\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dcf\u0db0\u0dca\u0dba\u0dba\u0db1\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0dc4\u0dcf \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc4\u0dbb\u0dc4\u0dcf \u0dc0\u0dbb\u0dca\u0d9c \u0d9a\u0dc5 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1; \u0d91\u0db1\u0db8\u0dca \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0dbd\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dcf\u0db0\u0dca\u0dba\u0dba\u0db1\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create buffers to store exponential moving averages of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba <span translate=no>_^_0_^_</span> \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0\u0dba\u0dda on \u0dcf\u0dad\u0dd3\u0dba \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba\u0db1\u0dca \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db6\u0dc6\u0dbb \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Create parameters for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for scale and shift </p>\n": "<p>\u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0dc3\u0dc4 \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 <span translate=no>_^_1_^_</span> \u0dc3\u0db3\u0dc4\u0dcf <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Here's <a href=\"mnist.html\">the training code</a> and a notebook for training a CNN classifier that uses batch normalization for MNIST dataset.</p>\n": "<p>MNIST\u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc3\u0dd3\u0d91\u0db1\u0dca\u0d91\u0db1\u0dca \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 <a href=\"mnist.html\">\u0d9a\u0dda\u0dad\u0dba</a> \u0dc3\u0dc4 \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0db4\u0ddc\u0dad\u0d9a\u0dca \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Internal covariate shift will adversely affect training speed because the later layers (<span translate=no>_^_0_^_</span> in the above example) have to adapt to this shifted distribution.</p>\n": "<p>\u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb\u0d9a\u0ddd\u0dc0\u0dd2\u0da0\u0dbb\u0dda\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dc0\u0dda\u0d9c\u0dba\u0da7 \u0d85\u0dc4\u0dd2\u0dad\u0d9a\u0dbb \u0dbd\u0dd9\u0dc3 \u0db6\u0dbd\u0db4\u0dcf\u0db1\u0dd4 \u0d87\u0dad, \u0db8\u0db1\u0dca\u0daf \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc3\u0dca\u0dae\u0dbb (\u0d89\u0dc4\u0dad<span translate=no>_^_0_^_</span> \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0dda) \u0db8\u0dd9\u0db8 \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dc5 \u0dc0\u0dca\u0dba\u0dcf\u0db4\u0dca\u0dad\u0dd2\u0dba\u0da7 \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0dc0\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4 \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dd2. </p>\n",
|
||||
"<p>It is known that whitening improves training speed and convergence. <em>Whitening</em> is linearly transforming inputs to have zero mean, unit variance, and be uncorrelated.</p>\n": "<p>\u0d91\u0dbawhitening \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dc0\u0dda\u0d9c\u0dba \u0dc4\u0dcf \u0d85\u0db7\u0dd2\u0dc3\u0dbb\u0dab\u0dba \u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1 \u0db6\u0dc0 \u0d9a\u0dc0\u0dd4\u0dbb\u0dd4\u0dad\u0dca \u0dc4\u0ddc\u0db3\u0dd2\u0db1\u0dca \u0daf\u0db1\u0dca\u0db1\u0dcf \u0d9a\u0dbb\u0dd4\u0dab\u0d9a\u0dd2. <em>\u0dc3\u0dd4\u0daf\u0dd4\u0d9a\u0dd2\u0dbb\u0dd3\u0db8</em> \u0dba\u0db1\u0dd4 \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba, \u0d92\u0d9a\u0d9a \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dc3\u0dc4 \u0dc3\u0dc4\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0 \u0db1\u0ddc\u0dc0\u0db1 \u0dbd\u0dd9\u0dc3 \u0dba\u0dd9\u0daf\u0dc0\u0dd4\u0db8\u0dca \u0dbb\u0dda\u0d9b\u0dd3\u0dba\u0dc0 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2. </p>\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u0db8\u0dd4\u0dbd\u0dca\u0dc4\u0dd0\u0da9\u0dba \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u200d\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Normalizing each feature to zero mean and unit variance could affect what the layer can represent. As an example paper illustrates that, if the inputs to a sigmoid are normalized most of it will be within <span translate=no>_^_0_^_</span> range where the sigmoid is linear. To overcome this each feature is scaled and shifted by two trained parameters <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span>. <span translate=no>_^_3_^_</span> where <span translate=no>_^_4_^_</span> is the output of the batch normalization layer.</p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d85\u0d82\u0d9c\u0dba \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba\u0da7 \u0dc4\u0dcf \u0d92\u0d9a\u0d9a \u0dc0\u0dd2\u0da0\u0dbd\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba\u0da7 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0db1\u0dd2\u0dba\u0ddd\u0da2\u0db1\u0dba \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0daf\u0dd9\u0dba\u0da7 \u0db6\u0dbd\u0db4\u0dcf\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0db6\u0dc0 \u0db4\u0dd9\u0db1\u0dca\u0db1\u0dd4\u0db8\u0dca, \u0d91\u0dba sigmoid \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0dba\u0dd9\u0daf\u0dc0\u0dd4\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db1\u0db8\u0dca, \u0db6\u0ddc\u0dc4\u0ddd \u0d91\u0dba \u0dc3\u0dd2\u0d9c\u0dca\u0db8\u0ddd\u0dba\u0dd2\u0da9\u0dca \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d9a\u0ddc\u0dc4\u0dd9\u0daf <span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0dc3\u0dba \u0dad\u0dd4\u0dc5 \u0dc0\u0db1\u0dd4 \u0d87\u0dad. \u0db8\u0dd9\u0dba \u0da2\u0dba \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dd1\u0db8 \u0d85\u0d82\u0d9c\u0dba\u0d9a\u0dca\u0db8 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca <span translate=no>_^_1_^_</span> \u0daf\u0dd9\u0d9a\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0dd0\u0db6\u0dda <span translate=no>_^_2_^_</span>. <span translate=no>_^_3_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba <span translate=no>_^_4_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n",
|
||||
"<p>Normalizing outside the gradient computation using pre-computed (detached) means and variances doesn't work. For instance. (ignoring variance), let <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> is a trained bias and <span translate=no>_^_3_^_</span> is an outside gradient computation (pre-computed constant).</p>\n": "<p>\u0db4\u0dd6\u0dbb\u0dca\u0dc0\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf (\u0dc0\u0dd9\u0db1\u0dca\u0dc0\u0dd6) \u0db8\u0dcf\u0db0\u0dca\u0dba\u0dba\u0db1\u0dca \u0dc3\u0dc4 \u0dc0\u0dd2\u0da0\u0dbd\u0dca\u0dba\u0dba\u0db1\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd2\u0db1\u0dca \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda \u0d9c\u0dab\u0db1\u0dba \u0db4\u0dd2\u0da7\u0dad \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3. (\u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0db1\u0ddc\u0dc3\u0dbd\u0d9a\u0dcf \u0dc4\u0dd0\u0dbb\u0dd3\u0db8), \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0\u0d9a\u0dca <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf \u0dba\u0db1\u0dca\u0db1 \u0dc3\u0dc4 <span translate=no>_^_3_^_</span> \u0db4\u0dd2\u0da7\u0dad \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dca (\u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db1\u0dd2\u0dba\u0dad\u0dba) \u0dc0\u0dda. <span translate=no>_^_2_^_</span> </p>\n",
|
||||
"<p>Note that <span translate=no>_^_0_^_</span> has no effect on <span translate=no>_^_1_^_</span>. Therefore, <span translate=no>_^_2_^_</span> will increase or decrease based <span translate=no>_^_3_^_</span>, and keep on growing indefinitely in each training update. The paper notes that similar explosions happen with variances.</p>\n": "<p>\u0d9a\u0dd2\u0dc3\u0dd2\u0daf\u0dd4\u0db6\u0dbd\u0db4\u0dd1\u0db8\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0d87\u0dad\u0dd2 \u0db1\u0ddc\u0dc0\u0db1 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span>. \u0d92 \u0db1\u0dd2\u0dc3\u0dcf, \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd0\u0da9\u0dd2 \u0dc4\u0ddd \u0d85\u0da9\u0dd4 <span translate=no>_^_2_^_</span> \u0dc0\u0db1\u0dd4 \u0d87\u0dad <span translate=no>_^_3_^_</span>, \u0dc3\u0dc4 \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0daf\u0dd2\u0db1 \u0db1\u0dd2\u0dba\u0db8\u0dba\u0d9a\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2\u0dc0 \u0dc0\u0dbb\u0dca\u0db0\u0db1\u0dba \u0daf\u0dd2\u0d9c\u0da7\u0db8. \u0dc3\u0db8\u0dcf\u0db1 \u0db4\u0dd2\u0db4\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0d9a\u0db8\u0dca \u0dc3\u0db8\u0d9f \u0dc3\u0dd2\u0daf\u0dd4\u0dc0\u0db1 \u0db6\u0dc0 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>Note that when applying batch normalization after a linear transform like <span translate=no>_^_0_^_</span> the bias parameter <span translate=no>_^_1_^_</span> gets cancelled due to normalization. So you can and should omit bias parameter in linear transforms right before the batch normalization.</p>\n": "<p>\u0dbb\u0dda\u0d9b\u0dd3\u0dba\u0db4\u0dbb\u0dd2\u0dab\u0dcf\u0db8\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dba\u0ddc\u0daf\u0db1 \u0dc0\u0dd2\u0da7 \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba <span translate=no>_^_1_^_</span> \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc4\u0dda\u0dad\u0dd4\u0dc0\u0dd9\u0db1\u0dca \u0d85\u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0dc0\u0db1 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1. <span translate=no>_^_0_^_</span> \u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0da7 \u0db4\u0dd9\u0dbb \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0dda \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0d94\u0db6\u0da7 \u0db8\u0d9f \u0dc4\u0dd0\u0dbb\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4\u0dba. </p>\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dd0\u0dc0\u0dad\u0dc4\u0dd0\u0da9\u0d9c\u0dc3\u0dca\u0dc0\u0dcf \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u0db8\u0dd4\u0dbd\u0dca\u0db4\u0dd2\u0da7\u0db4\u0dad\u0da7 \u0db1\u0dd0\u0dc0\u0dad \u0dc4\u0dd0\u0da9\u0d9c\u0dc3\u0dca\u0dc0\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0db4\u0dd0\u0db8\u0dd2\u0dab\u0dd3\u0db8 </p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c\u0d9c\u0dab\u0db1 \u0dc3\u0db8\u0dcf\u0db1 \u0db6\u0dc0 \u0dad\u0dc4\u0dc0\u0dd4\u0dbb\u0dd4 \u0d9a\u0dbb \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0db1\u0dd3\u0db4\u0dcf\u0dbb\u0d9a\u0dca\u0dc2\u0dcf\u0dc0 \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0dc3\u0dc4 \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>The paper defines <em>Internal Covariate Shift</em> as the change in the distribution of network activations due to the change in network parameters during training. For example, let's say there are two layers <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>. During the beginning of the training <span translate=no>_^_2_^_</span> outputs (inputs to <span translate=no>_^_3_^_</span>) could be in distribution <span translate=no>_^_4_^_</span>. Then, after some training steps, it could move to <span translate=no>_^_5_^_</span>. This is <em>internal covariate shift</em>.</p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0da2\u0dcf\u0dbd \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0dc0\u0dd3\u0db8 \u0dc4\u0dda\u0dad\u0dd4\u0dc0\u0dd9\u0db1\u0dca \u0da2\u0dcf\u0dbd \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0db6\u0dd9\u0daf\u0dcf \u0dc4\u0dd0\u0dbb\u0dd3\u0db8\u0dda \u0dc0\u0dd9\u0db1\u0dc3 \u0dbd\u0dd9\u0dc3 <em>\u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb \u0d9a\u0ddd\u0dc0\u0dbb\u0dd3\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0</em> \u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3, \u0dc3\u0dca\u0dae\u0dbb \u0daf\u0dd9\u0d9a\u0d9a\u0dca \u0d87\u0dad\u0dd2 \u0db6\u0dc0 <span translate=no>_^_0_^_</span> \u0d9a\u0dd2\u0dba\u0db8\u0dd4 <span translate=no>_^_1_^_</span>. \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7\u0dba\u0dda \u0daf\u0dd3 ( <span translate=no>_^_2_^_</span> \u0dba\u0dd9\u0daf\u0dc0\u0dd4\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 <span translate=no>_^_3_^_</span>) \u0db6\u0dd9\u0daf\u0dcf\u0dc4\u0dd0\u0dbb\u0dd3\u0db8\u0dda \u0dc0\u0dd2\u0dba \u0dc4\u0dd0\u0d9a\u0dd2 <span translate=no>_^_4_^_</span>. \u0dc3\u0db8\u0dc4\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0db4\u0dc3\u0dd4\u0dc0, \u0d91\u0dba \u0dc0\u0dd9\u0dad \u0d9c\u0db8\u0db1\u0dca \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba <span translate=no>_^_5_^_</span>. \u0db8\u0dd9\u0dba <em>\u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb \u0d9a\u0ddd\u0dc0\u0dd2\u0dbb\u0dda\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0\u0d9a\u0dd2</em>. </p>\n",
|
||||
"<p>The paper introduces a simplified version which they call <em>Batch Normalization</em>. First simplification is that it normalizes each feature independently to have zero mean and unit variance: <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span>-dimensional input.</p>\n": "<p>\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2\u0dc3\u0dbb\u0dbd \u0d9a\u0dc5 \u0d85\u0db1\u0dd4\u0dc0\u0dcf\u0daf\u0dba\u0d9a\u0dca \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0d85\u0dad\u0dbb \u0d91\u0dba <em>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</em>\u0dbd\u0dd9\u0dc3 \u0dc4\u0dd0\u0db3\u0dd2\u0db1\u0dca\u0dc0\u0dda. \u0db4\u0dc5\u0db8\u0dd4 \u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db1\u0db8\u0dca, \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0dc3\u0dc4 \u0d92\u0d9a\u0d9a \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dad\u0dd2\u0db6\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d85\u0d82\u0d9c\u0dba \u0dc3\u0dca\u0dc0\u0dcf\u0db0\u0dd3\u0db1\u0dc0 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2: <span translate=no>_^_2_^_</span>-dimensional <span translate=no>_^_1_^_</span> \u0d86\u0daf\u0dcf\u0db1\u0dba <span translate=no>_^_0_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n",
|
||||
"<p>The second simplification is to use estimates of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> from the mini-batch for normalization; instead of calculating the mean and variance across the whole dataset.</p>\n": "<p>\u0daf\u0dd9\u0dc0\u0db1\u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db1\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dd4\u0da9\u0dcf <span translate=no>_^_1_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dd9\u0db1\u0dca \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba <span translate=no>_^_0_^_</span> \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2; \u0db8\u0dd4\u0dc5\u0dd4 \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db4\u0dd4\u0dbb\u0dcf\u0db8 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7. </p>\n",
|
||||
"<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch Normalization from paper <a href=\"https://arxiv.org/abs/1502.03167\">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc0\u0dd9\u0dad\u0dd2\u0db1\u0dca <a href=\"https://arxiv.org/abs/1502.03167\">\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8: \u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb \u0d9a\u0ddd\u0dc0\u0dbb\u0dd2\u0dba\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 \u0d85\u0da9\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0d9c\u0dd0\u0db9\u0dd4\u0dbb\u0dd4 \u0da2\u0dcf\u0dbd \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca</a> \u0d9a\u0dd2\u0dbb\u0dd3\u0db8. </p>\n",
|
||||
"<p>Update exponential moving averages </p>\n": "<p>\u0d9d\u0dcf\u0dad\u0dd3\u0dba\u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 </p>\n",
|
||||
"<p>Use exponential moving averages as estimates </p>\n": "<p>\u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0dbd\u0dd9\u0dc3 \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dbd\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>We need to know <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in order to perform the normalization. So during inference, you either need to go through the whole (or part of) dataset and find the mean and variance, or you can use an estimate calculated during training. The usual practice is to calculate an exponential moving average of mean and variance during the training phase and use that for inference.</p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4 \u0daf\u0dd0\u0db1\u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4 <span translate=no>_^_0_^_</span> \u0d85\u0dad\u0dbb <span translate=no>_^_1_^_</span> \u0d91\u0dba \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0dba. \u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d85\u0db1\u0dd4\u0db8\u0dcf\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3, \u0d94\u0db6\u0da7 \u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab (\u0dc4\u0ddd \u0d9a\u0ddc\u0da7\u0dc3\u0d9a\u0dca) \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0d9c\u0ddc\u0dc3\u0dca \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dc3\u0ddc\u0dba\u0dcf \u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4\u0dba, \u0db1\u0dd0\u0dad\u0dc4\u0ddc\u0dad\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. \u0dc3\u0dd4\u0db4\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc0\u0db1\u0dca\u0db1\u0dda \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d85\u0dc0\u0db0\u0dd2\u0dba\u0dda\u0daf\u0dd3 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0\u0dba\u0dda on \u0dcf\u0dad\u0dd3\u0dba \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0d85\u0db1\u0dd4\u0db8\u0dcf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2. </p>\n",
|
||||
"<p>We will calculate the mini-batch mean and variance if we are in training mode or if we have not tracked exponential moving averages </p>\n": "<p>\u0d85\u0db4\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db8\u0dcf\u0daf\u0dd2\u0dbd\u0dd2\u0dba\u0dda \u0dc3\u0dd2\u0da7\u0dd3 \u0db1\u0db8\u0dca \u0dc4\u0ddd on \u0dcf\u0dad\u0dd3\u0dba \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba\u0db1\u0dca \u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0d9a\u0dbb \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db1\u0db8\u0dca \u0d85\u0db4\u0dd2 \u0d9a\u0dd4\u0da9\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0dc3\u0dc4 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dd9\u0db8\u0dd4 </p>\n",
|
||||
"<p>Whitening is computationally expensive because you need to de-correlate and the gradients must flow through the full whitening calculation.</p>\n": "<p>\u0d94\u0db6\u0daf-\u0dc3\u0dc4\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0db1\u0dd2\u0dc3\u0dcf whitening \u0db4\u0dbb\u0dd2\u0d9c\u0dab\u0d9a\u0db8\u0dba \u0db8\u0dd2\u0dbd \u0d85\u0db0\u0dd2\u0d9a \u0dc0\u0db1 \u0d85\u0dad\u0dbb, \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab whitening \u0d9c\u0dab\u0db1\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0d9c\u0dbd\u0dcf \u0dba\u0dd4\u0dad\u0dd4\u0dba. </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_5_^_</span> is whether to scale and shift the normalized value </li>\n<li><span translate=no>_^_6_^_</span> is whether to calculate the moving averages or mean and variance</li></ul>\n<p>We've tried to use the same names for arguments as PyTorch <span translate=no>_^_7_^_</span> implementation.</p>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0d86\u0daf\u0dcf\u0db1\u0dba\u0dda \u0d87\u0dad\u0dd2 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_1_^_</span> \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc3\u0dca\u0dae\u0dcf\u0dba\u0dd2\u0dad\u0dcf\u0dc0 <span translate=no>_^_3_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0dc0\u0dda <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_4_^_</span> \u0d9d\u0dcf\u0dad\u0dd3\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0d9c\u0db1\u0dd2\u0db8\u0dd2\u0db1\u0dca \u0d9c\u0db8\u0dca\u0dba\u0dad\u0dcf\u0dc0 \u0dc0\u0dda </li>\n<li><span translate=no>_^_5_^_</span> \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dc5 \u0d85\u0d9c\u0dba \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1\u0dba\u0dd2 </li>\n<li><span translate=no>_^_6_^_</span> \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0dc4\u0ddd \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1\u0dba\u0dd2</li></ul>\n<p>PyTorch <span translate=no>_^_7_^_</span> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc0\u0dd0\u0db1\u0dd2 \u0dad\u0dbb\u0dca\u0d9a \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0db8 \u0db1\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0db4\u0dd2 \u0d8b\u0dad\u0dca\u0dc3\u0dcf\u0dc4 \u0d9a\u0dbb \u0d87\u0dad\u0dca\u0dad\u0dd9\u0db8\u0dd4. </p>\n",
|
||||
"A PyTorch implementation/tutorial of batch normalization.": "\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba.",
|
||||
"Batch Normalization": "\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"<h1>Batch Normalization</h1>\n": "<h1>\u6279\u91cf\u6807\u51c6\u5316</h1>\n",
|
||||
"<h2>Batch Normalization Layer</h2>\n<p>Batch normalization layer <span translate=no>_^_0_^_</span> normalizes the input <span translate=no>_^_1_^_</span> as follows:</p>\n<p>When input <span translate=no>_^_2_^_</span> is a batch of image representations, where <span translate=no>_^_3_^_</span> is the batch size, <span translate=no>_^_4_^_</span> is the number of channels, <span translate=no>_^_5_^_</span> is the height and <span translate=no>_^_6_^_</span> is the width. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span>. <span translate=no>_^_9_^_</span></p>\n<p>When input <span translate=no>_^_10_^_</span> is a batch of embeddings, where <span translate=no>_^_11_^_</span> is the batch size and <span translate=no>_^_12_^_</span> is the number of features. <span translate=no>_^_13_^_</span> and <span translate=no>_^_14_^_</span>. <span translate=no>_^_15_^_</span></p>\n<p>When input <span translate=no>_^_16_^_</span> is a batch of a sequence embeddings, where <span translate=no>_^_17_^_</span> is the batch size, <span translate=no>_^_18_^_</span> is the number of features, and <span translate=no>_^_19_^_</span> is the length of the sequence. <span translate=no>_^_20_^_</span> and <span translate=no>_^_21_^_</span>. <span translate=no>_^_22_^_</span></p>\n": "<h2>\u6279\u91cf\u5f52\u4e00\u5316\u5c42</h2>\n<p>\u6279\u91cf\u5f52\u4e00\u5316\u5c42\u5c06\u8f93\u5165<span translate=no>_^_0_^_</span>\u5f52\u4e00\u5316\uff0c<span translate=no>_^_1_^_</span>\u5982\u4e0b\u6240\u793a\uff1a</p>\n<p>\u5f53\u8f93\u5165<span translate=no>_^_2_^_</span>\u662f\u4e00\u6279\u56fe\u50cf\u8868\u793a\u65f6\uff0c\u5176\u4e2d<span translate=no>_^_3_^_</span>\u662f\u6279\u6b21\u5927\u5c0f\uff0c<span translate=no>_^_4_^_</span>\u662f\u901a\u9053\u6570\uff0c<span translate=no>_^_5_^_</span>\u662f\u9ad8\u5ea6\u548c<span translate=no>_^_6_^_</span>\u662f\u5bbd\u5ea6\u3002<span translate=no>_^_7_^_</span>\u548c<span translate=no>_^_8_^_</span>\u3002<span translate=no>_^_9_^_</span></p>\n<p>\u5f53\u8f93\u5165<span translate=no>_^_10_^_</span>\u662f\u4e00\u6279\u5d4c\u5165\u65f6\uff0c\u5176\u4e2d<span translate=no>_^_11_^_</span>\u662f\u6279\u6b21\u5927\u5c0f\uff0c<span translate=no>_^_12_^_</span>\u662f\u8981\u7d20\u7684\u6570\u91cf\u3002<span translate=no>_^_13_^_</span>\u548c<span translate=no>_^_14_^_</span>\u3002<span translate=no>_^_15_^_</span></p>\n<p>\u5f53\u8f93\u5165<span translate=no>_^_16_^_</span>\u662f\u4e00\u6279\u5e8f\u5217\u5d4c\u5165\u65f6\uff0c\u5176\u4e2d<span translate=no>_^_17_^_</span>\u662f\u6279\u6b21\u5927\u5c0f\uff0c<span translate=no>_^_18_^_</span>\u662f\u8981\u7d20\u7684\u6570\u91cf\uff0c<span translate=no>_^_19_^_</span>\u662f\u987a\u5e8f\u3002<span translate=no>_^_20_^_</span>\u548c<span translate=no>_^_21_^_</span>\u3002<span translate=no>_^_22_^_</span></p>\n",
|
||||
"<h2>Inference</h2>\n": "<h2>\u63a8\u65ad</h2>\n",
|
||||
"<h2>Normalization</h2>\n": "<h2>\u89c4\u8303\u5316</h2>\n",
|
||||
"<h3>Batch Normalization</h3>\n": "<h3>\u6279\u91cf\u6807\u51c6\u5316</h3>\n",
|
||||
"<h3>Internal Covariate Shift</h3>\n": "<h3>\u5185\u90e8\u534f\u53d8\u91cf\u79fb\u4f4d</h3>\n",
|
||||
"<h3>Normalizing outside gradient computation doesn't work</h3>\n": "<h3>\u5bf9\u5916\u90e8\u68af\u5ea6\u8ba1\u7b97\u8fdb\u884c\u5f52\u4e00\u5316\u4e0d\u8d77\u4f5c\u7528</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u5f20\u91cf<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u8868\u793a\u4efb\u610f\u6570\u91cf\uff08\u53ef\u80fd\u4e3a 0\uff09\u7684\u7ef4\u5ea6\u3002\u4f8b\u5982\uff0c\u5728\u56fe\u50cf\uff082D\uff09\u5377\u79ef\u4e2d\uff0c\u8fd9\u5c06\u662f<span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p> Simple test</p>\n": "<p>\u7b80\u5355\u6d4b\u8bd5</p>\n",
|
||||
"<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<p>Batch normalization also makes the back propagation invariant to the scale of the weights and empirically it improves generalization, so it has regularization effects too.</p>\n": "<p>\u6279\u91cf\u5f52\u4e00\u5316\u8fd8\u4f7f\u53cd\u5411\u4f20\u64ad\u4e0e\u6743\u91cd\u7684\u6bd4\u4f8b\u4fdd\u6301\u4e0d\u53d8\uff0c\u4ece\u7ecf\u9a8c\u4e0a\u8bb2\uff0c\u5b83\u6539\u5584\u4e86\u6cdb\u5316\uff0c\u56e0\u6b64\u5b83\u4e5f\u5177\u6709\u6b63\u5219\u5316\u6548\u679c\u3002</p>\n",
|
||||
"<p>By stabilizing the distribution, batch normalization minimizes the internal covariate shift.</p>\n": "<p>\u901a\u8fc7\u7a33\u5b9a\u5206\u5e03\uff0c\u6279\u91cf\u5f52\u4e00\u5316\u53ef\u4ee5\u6700\u5927\u9650\u5ea6\u5730\u51cf\u5c11\u5185\u90e8\u534f\u53d8\u91cf\u504f\u79fb\u3002</p>\n",
|
||||
"<p>Calculate the mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u7b2c\u4e00\u7ef4\u548c\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684\u5e73\u5747\u503c\uff1b\u5373\u6bcf\u4e2a\u8981\u7d20\u7684\u5747\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u7b2c\u4e00\u7ef4\u548c\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684\u5747\u65b9\u503c\uff1b\u5373\u6bcf\u4e2a\u8981\u7d20\u7684\u5747\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create buffers to store exponential moving averages of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> </p>\n": "<p>\u521b\u5efa\u7f13\u51b2\u533a\u4ee5\u5b58\u50a8\u5747\u503c<span translate=no>_^_0_^_</span>\u548c\u65b9\u5dee\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Create parameters for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for scale and shift </p>\n": "<p><span translate=no>_^_1_^_</span>\u4e3a\u7f29\u653e<span translate=no>_^_0_^_</span>\u548c\u79fb\u4f4d\u521b\u5efa\u53c2\u6570</p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u83b7\u53d6\u6279\u6b21\u5927\u5c0f</p>\n",
|
||||
"<p>Here's <a href=\"mnist.html\">the training code</a> and a notebook for training a CNN classifier that uses batch normalization for MNIST dataset.</p>\n": "<p><a href=\"mnist.html\">\u4ee5\u4e0b\u662f\u8bad\u7ec3\u4ee3\u7801</a>\u548c\u7528\u4e8e\u8bad\u7ec3 CNN \u5206\u7c7b\u5668\u7684\u7b14\u8bb0\u672c\uff0c\u8be5\u5206\u7c7b\u5668\u4f7f\u7528 MNIST \u6570\u636e\u96c6\u7684\u6279\u91cf\u5f52\u4e00\u5316\u3002</p>\n",
|
||||
"<p>Internal covariate shift will adversely affect training speed because the later layers (<span translate=no>_^_0_^_</span> in the above example) have to adapt to this shifted distribution.</p>\n": "<p>\u5185\u90e8\u534f\u53d8\u91cf\u504f\u79fb\u5c06\u5bf9\u8bad\u7ec3\u901f\u5ea6\u4ea7\u751f\u4e0d\u5229\u5f71\u54cd\uff0c\u56e0\u4e3a\u540e\u9762\u7684\u56fe\u5c42\uff08\u5728\u4e0a\u9762\u7684\u4f8b\u5b50<span translate=no>_^_0_^_</span>\u4e2d\uff09\u5fc5\u987b\u9002\u5e94\u8fd9\u79cd\u504f\u79fb\u5206\u5e03\u3002</p>\n",
|
||||
"<p>It is known that whitening improves training speed and convergence. <em>Whitening</em> is linearly transforming inputs to have zero mean, unit variance, and be uncorrelated.</p>\n": "<p>\u4f17\u6240\u5468\u77e5\uff0c\u7f8e\u767d\u53ef\u4ee5\u63d0\u9ad8\u8bad\u7ec3\u901f\u5ea6\u548c\u6536\u655b\u6027\u3002<em>\u7f8e\u767d</em>\u662f\u5c06\u8f93\u5165\u8fdb\u884c\u7ebf\u6027\u53d8\u6362\uff0c\u4f7f\u5176\u5747\u503c\u4e3a\u96f6\u3001\u5355\u4f4d\u65b9\u5dee\u4e14\u4e0d\u76f8\u5173\u3002</p>\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u4fdd\u6301\u539f\u59cb\u5f62\u72b6</p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u89c4\u8303\u5316<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Normalizing each feature to zero mean and unit variance could affect what the layer can represent. As an example paper illustrates that, if the inputs to a sigmoid are normalized most of it will be within <span translate=no>_^_0_^_</span> range where the sigmoid is linear. To overcome this each feature is scaled and shifted by two trained parameters <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span>. <span translate=no>_^_3_^_</span> where <span translate=no>_^_4_^_</span> is the output of the batch normalization layer.</p>\n": "<p>\u5c06\u6bcf\u4e2a\u8981\u7d20\u5f52\u4e00\u5316\u4e3a\u96f6\u5747\u503c\u548c\u5355\u4f4d\u65b9\u5dee\u53ef\u80fd\u4f1a\u5f71\u54cd\u56fe\u5c42\u53ef\u4ee5\u8868\u793a\u7684\u5185\u5bb9\u3002\u4f5c\u4e3a\u793a\u4f8b\u8bba\u6587\u8bf4\u660e\uff0c\u5982\u679csigmoid\u7684\u8f93\u5165\u88ab\u5f52\u4e00\u5316\uff0c\u5219\u5927\u90e8\u5206\u5c06\u5728sigmoid\u4e3a\u7ebf\u6027\u7684<span translate=no>_^_0_^_</span>\u8303\u56f4\u5185\u3002\u4e3a\u4e86\u514b\u670d\u8fd9\u4e2a\u95ee\u9898\uff0c\u6bcf\u4e2a\u7279\u5f81\u90fd\u901a\u8fc7\u4e24\u4e2a\u7ecf\u8fc7\u8bad\u7ec3\u7684\u53c2\u6570\u8fdb\u884c\u7f29\u653e<span translate=no>_^_1_^_</span>\u548c\u79fb\u52a8<span translate=no>_^_2_^_</span>\u3002<span translate=no>_^_3_^_</span>\u5176\u4e2d<span translate=no>_^_4_^_</span>\u662f\u6279\u91cf\u5f52\u4e00\u5316\u5c42\u7684\u8f93\u51fa\u3002</p>\n",
|
||||
"<p>Normalizing outside the gradient computation using pre-computed (detached) means and variances doesn't work. For instance. (ignoring variance), let <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> is a trained bias and <span translate=no>_^_3_^_</span> is an outside gradient computation (pre-computed constant).</p>\n": "<p>\u4f7f\u7528\u9884\u5148\u8ba1\u7b97\uff08\u5206\u79bb\uff09\u5747\u503c\u548c\u65b9\u5dee\u5728\u68af\u5ea6\u8ba1\u7b97\u4e4b\u5916\u8fdb\u884c\u5f52\u4e00\u5316\u4e0d\u8d77\u4f5c\u7528\u3002\u4f8b\u5982\u3002\uff08\u5ffd\u7565\u65b9\u5dee\uff09<span translate=no>_^_0_^_</span>\uff0c\u8ba9 wher<span translate=no>_^_1_^_</span> e an<span translate=no>_^_2_^_</span> d \u662f\u4e00\u4e2a\u8bad\u7ec3\u8fc7\u7684\u504f\u5dee\uff0c<span translate=no>_^_3_^_</span>\u662f\u5916\u90e8\u68af\u5ea6\u8ba1\u7b97\uff08\u9884\u5148\u8ba1\u7b97\u7684\u5e38\u91cf\uff09\u3002</p>\n",
|
||||
"<p>Note that <span translate=no>_^_0_^_</span> has no effect on <span translate=no>_^_1_^_</span>. Therefore, <span translate=no>_^_2_^_</span> will increase or decrease based <span translate=no>_^_3_^_</span>, and keep on growing indefinitely in each training update. The paper notes that similar explosions happen with variances.</p>\n": "<p>\u8bf7\u6ce8\u610f<span translate=no>_^_0_^_</span>\uff0c\u8fd9\u5bf9<span translate=no>_^_1_^_</span>\u3002\u56e0\u6b64\uff0c<span translate=no>_^_2_^_</span>\u5c06\u5728\u6bcf\u6b21\u8bad\u7ec3\u66f4\u65b0\u4e2d\u589e\u52a0\u6216\u51cf\u5c11<span translate=no>_^_3_^_</span>\uff0c\u5e76\u4e14\u4f1a\u65e0\u9650\u671f\u5730\u589e\u957f\u3002\u8be5\u62a5\u6307\u51fa\uff0c\u7c7b\u4f3c\u7684\u7206\u70b8\u4f1a\u53d1\u751f\u5dee\u5f02\u3002</p>\n",
|
||||
"<p>Note that when applying batch normalization after a linear transform like <span translate=no>_^_0_^_</span> the bias parameter <span translate=no>_^_1_^_</span> gets cancelled due to normalization. So you can and should omit bias parameter in linear transforms right before the batch normalization.</p>\n": "<p>\u8bf7\u6ce8\u610f\uff0c\u5728\u7ebf\u6027\u53d8\u6362\u4e4b\u540e\u5e94\u7528\u6279\u91cf\u5f52\u4e00\u5316\u65f6\uff0c\u6bd4\u5982<span translate=no>_^_0_^_</span>\u504f\u7f6e\u53c2\u6570<span translate=no>_^_1_^_</span>\u4f1a\u56e0\u5f52\u4e00\u5316\u800c\u88ab\u53d6\u6d88\u3002\u56e0\u6b64\uff0c\u4f60\u53ef\u4ee5\u800c\u4e14\u5e94\u8be5\u5728\u6279\u91cf\u5f52\u4e00\u5316\u4e4b\u524d\u7701\u7565\u7ebf\u6027\u53d8\u6362\u4e2d\u7684\u504f\u7f6e\u53c2\u6570\u3002</p>\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u91cd\u5851\u6210<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u91cd\u5851\u4e3a\u539f\u59cb\u5f62\u72b6\u7136\u540e\u8fd4\u56de</p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u8fdb\u884c\u5065\u5168\u6027\u68c0\u67e5\u4ee5\u786e\u4fdd\u8981\u7d20\u6570\u91cf\u76f8\u540c</p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7f29\u653e\u548c\u79fb\u52a8<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The paper defines <em>Internal Covariate Shift</em> as the change in the distribution of network activations due to the change in network parameters during training. For example, let's say there are two layers <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>. During the beginning of the training <span translate=no>_^_2_^_</span> outputs (inputs to <span translate=no>_^_3_^_</span>) could be in distribution <span translate=no>_^_4_^_</span>. Then, after some training steps, it could move to <span translate=no>_^_5_^_</span>. This is <em>internal covariate shift</em>.</p>\n": "<p>\u672c\u6587\u5c06<em>\u5185\u90e8\u534f\u53d8\u91cf\u79fb\u4f4d</em>\u5b9a\u4e49\u4e3a\u8bad\u7ec3\u671f\u95f4\u7531\u4e8e\u7f51\u7edc\u53c2\u6570\u7684\u53d8\u5316\u800c\u5bfc\u81f4\u7684\u7f51\u7edc\u6fc0\u6d3b\u5206\u5e03\u7684\u53d8\u5316\u3002\u4f8b\u5982\uff0c\u5047\u8bbe\u6709\u4e24\u5c42<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span>\u3002\u5728\u57f9\u8bad\u5f00\u59cb\u65f6\uff0c\u53ef\u4ee5\u5206\u53d1<span translate=no>_^_2_^_</span>\u8f93\u51fa\uff08\u8f93\u5165<span translate=no>_^_3_^_</span>\uff09<span translate=no>_^_4_^_</span>\u3002\u7136\u540e\uff0c\u7ecf\u8fc7\u4e00\u4e9b\u8bad\u7ec3\u6b65\u9aa4\u540e\uff0c\u5b83\u53ef\u80fd\u4f1a\u79fb\u81f3<span translate=no>_^_5_^_</span>\u3002\u8fd9\u662f<em>\u5185\u90e8\u534f\u53d8\u91cf\u79fb\u4f4d</em>\u3002</p>\n",
|
||||
"<p>The paper introduces a simplified version which they call <em>Batch Normalization</em>. First simplification is that it normalizes each feature independently to have zero mean and unit variance: <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span>-dimensional input.</p>\n": "<p>\u672c\u6587\u4ecb\u7ecd\u4e86\u4e00\u4e2a\u7b80\u5316\u7248\u672c\uff0c\u4ed6\u4eec\u79f0\u4e4b\u4e3a<em>\u6279\u91cf\u89c4\u8303\u5316</em>\u3002\u9996\u5148\u7b80\u5316\u7684\u662f\uff0c\u5b83\u5c06\u6bcf\u4e2a\u8981\u7d20\u72ec\u7acb\u5f52\u4e00\u5316\uff0c\u4f7f\u5176\u5747\u503c\u548c\u5355\u4f4d\u65b9\u5dee\u4e3a\u96f6\uff1a<span translate=no>_^_0_^_</span>\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f<span translate=no>_^_2_^_</span>\u7ef4\u5ea6\u8f93\u5165\u3002</p>\n",
|
||||
"<p>The second simplification is to use estimates of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> from the mini-batch for normalization; instead of calculating the mean and variance across the whole dataset.</p>\n": "<p>\u7b2c\u4e8c\u79cd\u7b80\u5316\u65b9\u6cd5\u662f\u4f7f\u7528<span translate=no>_^_1_^_</span>\u6765\u81ea\u5fae\u578b\u6279\u6b21\u7684\u5747\u503c<span translate=no>_^_0_^_</span>\u548c\u65b9\u5dee\u7684\u4f30\u8ba1\u503c\u8fdb\u884c\u5f52\u4e00\u5316\uff1b\u800c\u4e0d\u662f\u8ba1\u7b97\u6574\u4e2a\u6570\u636e\u96c6\u7684\u5747\u503c\u548c\u65b9\u5dee\u3002</p>\n",
|
||||
"<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch Normalization from paper <a href=\"https://arxiv.org/abs/1502.03167\">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a>.</p>\n": "<p>\u8fd9\u662f <a href=\"https://pytorch.org\">PyTorch</a> \u4ece\u7eb8\u8d28\u6279\u91cf\u89c4\u8303\u5316\u4e2d<a href=\"https://arxiv.org/abs/1502.03167\">\u5b9e\u73b0\u6279\u91cf\u5f52\u4e00\u5316\uff1a\u901a\u8fc7\u51cf\u5c11\u5185\u90e8\u534f\u53d8\u91cf\u504f\u79fb\u52a0\u901f\u6df1\u5ea6\u7f51\u7edc\u8bad\u7ec3</a>\u3002</p>\n",
|
||||
"<p>Update exponential moving averages </p>\n": "<p>\u66f4\u65b0\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf</p>\n",
|
||||
"<p>Use exponential moving averages as estimates </p>\n": "<p>\u4f7f\u7528\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u4f5c\u4e3a\u4f30\u8ba1\u503c</p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6bcf\u4e2a\u8981\u7d20\u7684\u65b9\u5dee<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>We need to know <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in order to perform the normalization. So during inference, you either need to go through the whole (or part of) dataset and find the mean and variance, or you can use an estimate calculated during training. The usual practice is to calculate an exponential moving average of mean and variance during the training phase and use that for inference.</p>\n": "<p>\u6211\u4eec\u9700\u8981\u77e5\u9053 an<span translate=no>_^_0_^_</span> d<span translate=no>_^_1_^_</span> \u624d\u80fd\u6267\u884c\u89c4\u8303\u5316\u3002\u56e0\u6b64\uff0c\u5728\u63a8\u7406\u8fc7\u7a0b\u4e2d\uff0c\u60a8\u8981\u4e48\u9700\u8981\u904d\u5386\u6574\u4e2a\uff08\u6216\u90e8\u5206\uff09\u6570\u636e\u96c6\u5e76\u627e\u5230\u5747\u503c\u548c\u65b9\u5dee\uff0c\u8981\u4e48\u53ef\u4ee5\u4f7f\u7528\u8bad\u7ec3\u671f\u95f4\u8ba1\u7b97\u7684\u4f30\u8ba1\u503c\u3002\u901a\u5e38\u7684\u505a\u6cd5\u662f\u5728\u8bad\u7ec3\u9636\u6bb5\u8ba1\u7b97\u5747\u503c\u548c\u65b9\u5dee\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c\u7136\u540e\u5c06\u5176\u7528\u4e8e\u63a8\u65ad\u3002</p>\n",
|
||||
"<p>We will calculate the mini-batch mean and variance if we are in training mode or if we have not tracked exponential moving averages </p>\n": "<p>\u5982\u679c\u6211\u4eec\u5904\u4e8e\u8bad\u7ec3\u6a21\u5f0f\u6216\u8005\u6ca1\u6709\u8ddf\u8e2a\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c\u6211\u4eec\u5c06\u8ba1\u7b97\u5c0f\u6279\u6b21\u5747\u503c\u548c\u65b9\u5dee</p>\n",
|
||||
"<p>Whitening is computationally expensive because you need to de-correlate and the gradients must flow through the full whitening calculation.</p>\n": "<p>\u7f8e\u767d\u5728\u8ba1\u7b97\u4e0a\u5f88\u6602\u8d35\uff0c\u56e0\u4e3a\u4f60\u9700\u8981\u53bb\u5173\u8054\uff0c\u800c\u4e14\u68af\u5ea6\u5fc5\u987b\u901a\u8fc7\u5b8c\u6574\u7684\u7f8e\u767d\u8ba1\u7b97\u3002</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is the momentum in taking the exponential moving average </li>\n<li><span translate=no>_^_5_^_</span> is whether to scale and shift the normalized value </li>\n<li><span translate=no>_^_6_^_</span> is whether to calculate the moving averages or mean and variance</li></ul>\n<p>We've tried to use the same names for arguments as PyTorch <span translate=no>_^_7_^_</span> implementation.</p>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8f93\u5165\u4e2d\u7684\u8981\u7d20\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f<span translate=no>_^_2_^_</span>\uff0c<span translate=no>_^_3_^_</span>\u7528\u4e8e\u6570\u503c\u7a33\u5b9a\u6027</li>\n<li><span translate=no>_^_4_^_</span>\u662f\u53d6\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\u7684\u52a8\u91cf</li>\n<li><span translate=no>_^_5_^_</span>\u662f\u5426\u7f29\u653e\u548c\u79fb\u52a8\u5f52\u4e00\u5316\u503c</li>\n<li><span translate=no>_^_6_^_</span>\u662f\u8ba1\u7b97\u79fb\u52a8\u5e73\u5747\u7ebf\u8fd8\u662f\u5747\u503c\u548c\u65b9\u5dee</li></ul>\n<p>\u6211\u4eec\u5df2\u7ecf\u5c1d\u8bd5\u4f7f\u7528\u4e0e PyTorch<span translate=no>_^_7_^_</span> \u5b9e\u73b0\u76f8\u540c\u7684\u53c2\u6570\u540d\u79f0\u3002</p>\n",
|
||||
"A PyTorch implementation/tutorial of batch normalization.": "\u4e00\u4e2a\u5173\u4e8e\u6279\u91cf\u89c4\u8303\u5316\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002",
|
||||
"Batch Normalization": "\u6279\u91cf\u6807\u51c6\u5316"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Group Normalization</h1>\n": "<h1>\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u306e\u305f\u3081\u306e CIFAR10 \u5b9f\u9a13</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u30e2\u30c7\u30eb\u4f5c\u6210</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>CIFAR-10 \u5206\u985e\u7528\u306e VGG \u30e2\u30c7\u30eb</h3>\n<p><a href=\"../../experiments/cifar10.html\">\u3053\u308c\u306f\u4e00\u822c\u7684\u306a VGG \u30b9\u30bf\u30a4\u30eb\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306b\u7531\u6765\u3057\u307e\u3059</a>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u69cb\u6210\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3092\u5b9f\u884c\u3057\u307e\u3059</p>\n",
|
||||
"CIFAR10 Experiment to try Group Normalization": "\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u3092\u8a66\u3059\u305f\u3081\u306e CIFAR10 \u5b9f\u9a13",
|
||||
"This trains is a simple convolutional neural network that uses group normalization to classify CIFAR10 images.": "\u3053\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306f\u3001\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3057\u3066 CIFAR10 \u753b\u50cf\u3092\u5206\u985e\u3059\u308b\u5358\u7d14\u306a\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Group Normalization</h1>\n": "<h1>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf CIFAR10 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>CIFA-10\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf VGG \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h3>\n<p>\u0db8\u0dd9\u0dba <a href=\"../../experiments/cifar10.html\">\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba VGG \u0dc0\u0dd2\u0dbd\u0dcf\u0dc3\u0dd2\u0dad\u0dcf\u0dc0\u0dda \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0dd9\u0db1\u0dca</a>\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dd3. </p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"CIFAR10 Experiment to try Group Normalization": "\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d8b\u0dad\u0dca\u0dc3\u0dcf\u0dc4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf CIFAR10 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This trains is a simple convolutional neural network that uses group normalization to classify CIFAR10 images.": "\u0db8\u0dd9\u0db8 \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba CIFAR10 \u0dbb\u0dd6\u0db4 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc3\u0dbb\u0dbd \u0dc3\u0d82\u0dc0\u0dc4\u0db1 \u0dc3\u0dca\u0db1\u0dcf\u0dba\u0dd4\u0d9a \u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Group Normalization</h1>\n": "<h1>CIFAR10 \u7fa4\u5f52\u4e00\u5316\u5b9e\u9a8c</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u521b\u5efa\u6a21\u578b</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>\u7528\u4e8e CIFAR-10 \u5206\u7c7b\u7684 VGG \u6a21\u578b</h3>\n<p>\u8fd9\u6e90\u4e8e<a href=\"../../experiments/cifar10.html\">\u901a\u7528\u7684 VGG \u98ce\u683c\u67b6\u6784</a>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c\u5e76\u8fd0\u884c\u8bad\u7ec3\u5faa\u73af</p>\n",
|
||||
"CIFAR10 Experiment to try Group Normalization": "CIFAR10 \u5c1d\u8bd5\u7fa4\u5f52\u4e00\u5316\u7684\u5b9e\u9a8c",
|
||||
"This trains is a simple convolutional neural network that uses group normalization to classify CIFAR10 images.": "\u8be5\u5217\u8f66\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff0c\u5b83\u4f7f\u7528\u7fa4\u5f52\u4e00\u5316\u5bf9 CIFAR10 \u56fe\u50cf\u8fdb\u884c\u5206\u7c7b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"<h1>MNIST Experiment for Batch Normalization</h1>\n": "<h1>\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306e\u305f\u3081\u306e MNIST \u5b9f\u9a13</h1>\n",
|
||||
"<h3>Create model</h3>\n<p>We use <a href=\"../../experiments/mnist.html#MNISTConfigs\"><span translate=no>_^_0_^_</span></a> configurations and set a new function to calculate the model.</p>\n": "<h3>\u30e2\u30c7\u30eb\u4f5c\u6210</h3>\n<p><a href=\"../../experiments/mnist.html#MNISTConfigs\"><span translate=no>_^_0_^_</span></a>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3\u3092\u4f7f\u7528\u3057\u3001\u65b0\u3057\u3044\u95a2\u6570\u3092\u8a2d\u5b9a\u3057\u3066\u30e2\u30c7\u30eb\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Model definition</h3>\n": "<h3>\u30e2\u30c7\u30eb\u5b9a\u7fa9</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Batch normalization with 20 channels (output of convolution layer). The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>20 \u30c1\u30e3\u30cd\u30eb (\u7573\u307f\u8fbc\u307f\u5c64\u306e\u51fa\u529b) \u306b\u3088\u308b\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3002\u3053\u306e\u30ec\u30a4\u30e4\u30fc\u3078\u306e\u5165\u529b\u306f\u30b7\u30a7\u30a4\u30d7\u306b\u306a\u308a\u307e\u3059</p>\u3002<span translate=no>_^_0_^_</span>\n",
|
||||
"<p>Batch normalization with 50 channels. The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>50 \u30c1\u30e3\u30f3\u30cd\u30eb\u306e\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3002\u3053\u306e\u30ec\u30a4\u30e4\u30fc\u3078\u306e\u5165\u529b\u306f\u30b7\u30a7\u30a4\u30d7\u306b\u306a\u308a\u307e\u3059</p>\u3002<span translate=no>_^_0_^_</span>\n",
|
||||
"<p>Batch normalization with 500 channels (output of fully connected layer). The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>500 \u30c1\u30e3\u30cd\u30eb (\u5b8c\u5168\u63a5\u7d9a\u5c64\u306e\u51fa\u529b) \u306b\u3088\u308b\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3053\u306e\u30ec\u30a4\u30e4\u30fc\u3078\u306e\u5165\u529b\u306f\u30b7\u30a7\u30a4\u30d7\u306b\u306a\u308a\u307e\u3059</p>\u3002<span translate=no>_^_0_^_</span>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u69cb\u6210\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Note that we omit the bias parameter </p>\n": "<p>\u30d0\u30a4\u30a2\u30b9\u30d1\u30e9\u30e1\u30fc\u30bf\u306f\u7701\u7565\u3057\u3066\u3044\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3092\u5b9f\u884c\u3057\u307e\u3059</p>\n",
|
||||
"MNIST Experiment to try Batch Normalization": "\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092\u8a66\u3059\u305f\u3081\u306eMNIST\u5b9f\u9a13",
|
||||
"This trains is a simple convolutional neural network that uses batch normalization to classify MNIST digits.": "\u3053\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306f\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3057\u3066MNIST\u306e\u6570\u5b57\u3092\u5206\u985e\u3059\u308b\u5358\u7d14\u306a\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"<h1>MNIST Experiment for Batch Normalization</h1>\n": "<h1>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf MNIST \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n",
|
||||
"<h3>Create model</h3>\n<p>We use <a href=\"../../experiments/mnist.html#MNISTConfigs\"><span translate=no>_^_0_^_</span></a> configurations and set a new function to calculate the model.</p>\n": "<h3>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h3>\n<p>\u0d85\u0db4\u0dd2 <a href=\"../../experiments/mnist.html#MNISTConfigs\"><span translate=no>_^_0_^_</span></a> \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dbb \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dc0 \u0dc1\u0dca\u0dbb\u0dd2\u0dad\u0dba\u0d9a\u0dca \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<h3>Model definition</h3>\n": "<h3>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0d85\u0dbb\u0dca\u0dae \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd3\u0db8</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Batch normalization with 20 channels (output of convolution layer). The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf20 \u0d9a\u0dca \u0dc3\u0dc4\u0dd2\u0dad \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba (\u0dc3\u0d82\u0dc0\u0dc4\u0db1 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba). \u0db8\u0dd9\u0db8 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0da7 \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dc4\u0dd0\u0da9\u0dba \u0d87\u0dad <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Batch normalization with 50 channels. The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf50 \u0d9a\u0dca \u0dc3\u0db8\u0d9f \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8. \u0db8\u0dd9\u0db8 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0da7 \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dc4\u0dd0\u0da9\u0dba \u0d87\u0dad <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Batch normalization with 500 channels (output of fully connected layer). The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf500 \u0d9a\u0dca \u0dc3\u0db8\u0d9f \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 (\u0dc3\u0db8\u0dca\u0db4\u0dd4\u0dbb\u0dca\u0dab\u0dba\u0dd9\u0db1\u0dca\u0db8 \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba). \u0db8\u0dd9\u0db8 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0da7 \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dc4\u0dd0\u0da9\u0dba \u0d87\u0dad <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Note that we omit the bias parameter </p>\n": "<p>\u0d85\u0db4\u0dd2\u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0db8\u0d9f \u0dc4\u0dd0\u0dbb\u0dd2 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"MNIST Experiment to try Batch Normalization": "\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d8b\u0dad\u0dca\u0dc3\u0dcf\u0dc4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf MNIST \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This trains is a simple convolutional neural network that uses batch normalization to classify MNIST digits.": "\u0db8\u0dd9\u0db8 \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba MNIST \u0d89\u0dbd\u0d9a\u0dca\u0d9a\u0db8\u0dca \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc3\u0dbb\u0dbd \u0dc3\u0d82\u0dc0\u0dc4\u0db1 \u0dc3\u0dca\u0db1\u0dcf\u0dba\u0dd4\u0d9a \u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"<h1>MNIST Experiment for Batch Normalization</h1>\n": "<h1>\u6279\u91cf\u6807\u51c6\u5316\u7684 MNIST \u5b9e\u9a8c</h1>\n",
|
||||
"<h3>Create model</h3>\n<p>We use <a href=\"../../experiments/mnist.html#MNISTConfigs\"><span translate=no>_^_0_^_</span></a> configurations and set a new function to calculate the model.</p>\n": "<h3>\u521b\u5efa\u6a21\u578b</h3>\n<p>\u6211\u4eec\u4f7f\u7528<a href=\"../../experiments/mnist.html#MNISTConfigs\"><span translate=no>_^_0_^_</span></a>\u914d\u7f6e\u5e76\u8bbe\u7f6e\u4e00\u4e2a\u65b0\u51fd\u6570\u6765\u8ba1\u7b97\u6a21\u578b\u3002</p>\n",
|
||||
"<h3>Model definition</h3>\n": "<h3>\u578b\u53f7\u5b9a\u4e49</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Batch normalization with 20 channels (output of convolution layer). The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5177\u6709 20 \u4e2a\u901a\u9053\u7684\u6279\u91cf\u5f52\u4e00\u5316\uff08\u5377\u79ef\u5c42\u7684\u8f93\u51fa\uff09\u3002\u6b64\u56fe\u5c42\u7684\u8f93\u5165\u5c06\u5177\u6709\u5f62\u72b6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Batch normalization with 50 channels. The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528 50 \u4e2a\u901a\u9053\u8fdb\u884c\u6279\u91cf\u5f52\u4e00\u5316\u3002\u6b64\u56fe\u5c42\u7684\u8f93\u5165\u5c06\u5177\u6709\u5f62\u72b6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Batch normalization with 500 channels (output of fully connected layer). The input to this layer will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528 500 \u4e2a\u901a\u9053\u8fdb\u884c\u6279\u91cf\u5f52\u4e00\u5316\uff08\u5b8c\u5168\u8fde\u63a5\u5c42\u7684\u8f93\u51fa\uff09\u3002\u6b64\u56fe\u5c42\u7684\u8f93\u5165\u5c06\u5177\u6709\u5f62\u72b6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Note that we omit the bias parameter </p>\n": "<p>\u8bf7\u6ce8\u610f\uff0c\u6211\u4eec\u7701\u7565\u4e86 bias \u53c2\u6570</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c\u5e76\u8fd0\u884c\u8bad\u7ec3\u5faa\u73af</p>\n",
|
||||
"MNIST Experiment to try Batch Normalization": "\u5c1d\u8bd5\u6279\u91cf\u5f52\u4e00\u5316\u7684 MNIST \u5b9e\u9a8c",
|
||||
"This trains is a simple convolutional neural network that uses batch normalization to classify MNIST digits.": "\u8be5\u5217\u8f66\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff0c\u5b83\u4f7f\u7528\u6279\u91cf\u5f52\u4e00\u5316\u5bf9MNIST\u6570\u5b57\u8fdb\u884c\u5206\u7c7b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">Batch Normalization</a></h1>\n": "<h1><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">\u30d0\u30c3\u30c1\u6b63\u898f\u5316</a></h1>\n",
|
||||
"<h2>Inference</h2>\n": "<h2>\u63a8\u8ad6</h2>\n",
|
||||
"<h2>Normalization</h2>\n": "<h2>\u30ce\u30fc\u30de\u30e9\u30a4\u30bc\u30fc\u30b7\u30e7\u30f3</h2>\n",
|
||||
"<h3>Batch Normalization</h3>\n": "<h3>\u30d0\u30c3\u30c1\u6b63\u898f\u5316</h3>\n",
|
||||
"<h3>Internal Covariate Shift</h3>\n": "<h3>\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8</h3>\n",
|
||||
"<h3>Normalizing outside gradient computation doesn't work</h3>\n": "<h3>\u5916\u90e8\u52fe\u914d\u8a08\u7b97\u306e\u6b63\u898f\u5316\u306f\u6a5f\u80fd\u3057\u307e\u305b\u3093</h3>\n",
|
||||
"<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a> </p>\n": "<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<p>Batch normalization also makes the back propagation invariant to the scale of the weights and empirically it improves generalization, so it has regularization effects too.</p>\n": "<p>\u307e\u305f\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3067\u306f\u9006\u4f1d\u64ad\u304c\u91cd\u307f\u306e\u30b9\u30b1\u30fc\u30eb\u306b\u5bfe\u3057\u3066\u4e0d\u5909\u306b\u306a\u308a\u3001\u7d4c\u9a13\u7684\u306b\u30b8\u30a7\u30cd\u30e9\u30e9\u30a4\u30ba\u304c\u6539\u5584\u3055\u308c\u308b\u305f\u3081\u3001\u6b63\u5247\u5316\u52b9\u679c\u3082\u3042\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<p>By stabilizing the distribution, batch normalization minimizes the internal covariate shift.</p>\n": "<p>\u5206\u5e03\u3092\u5b89\u5b9a\u3055\u305b\u308b\u3053\u3068\u306b\u3088\u308a\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306f\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u3092\u6700\u5c0f\u9650\u306b\u6291\u3048\u307e\u3059\u3002</p>\n",
|
||||
"<p>Here's <a href=\"mnist.html\">the training code</a> and a notebook for training a CNN classifier that uses batch normalization for MNIST dataset.</p>\n": "<p>\u4ee5\u4e0b\u306f<a href=\"mnist.html\">\u3001MNIST \u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3059\u308b CNN \u5206\u985e\u5668\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3068\u30ce\u30fc\u30c8\u30d6\u30c3\u30af\u3067\u3059</a>\u3002</p>\n",
|
||||
"<p>Internal covariate shift will adversely affect training speed because the later layers (<span translate=no>_^_0_^_</span> in the above example) have to adapt to this shifted distribution.</p>\n": "<p>\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u306f\u3001\u5f8c\u306e\u5c64\uff08<span translate=no>_^_0_^_</span>\u4e0a\u306e\u4f8b\uff09\u304c\u3053\u306e\u30b7\u30d5\u30c8\u3057\u305f\u5206\u5e03\u306b\u9069\u5fdc\u3057\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u305f\u3081\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u901f\u5ea6\u306b\u60aa\u5f71\u97ff\u3092\u53ca\u307c\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>It is known that whitening improves training speed and convergence. <em>Whitening</em> is linearly transforming inputs to have zero mean, unit variance, and be uncorrelated.</p>\n": "<p>\u30db\u30ef\u30a4\u30c8\u30cb\u30f3\u30b0\u306f\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u30b9\u30d4\u30fc\u30c9\u3068\u30b3\u30f3\u30d0\u30fc\u30b8\u30a7\u30f3\u30b9\u3092\u5411\u4e0a\u3055\u305b\u308b\u3053\u3068\u304c\u77e5\u3089\u308c\u3066\u3044\u307e\u3059\u3002<em>\u30db\u30ef\u30a4\u30c8\u30cb\u30f3\u30b0\u3068\u306f</em>\u3001\u5165\u529b\u3092\u5e73\u5747\u304c\u30bc\u30ed\u3001\u5358\u4f4d\u5206\u6563\u3001\u7121\u76f8\u95a2\u306b\u306a\u308b\u3088\u3046\u306b\u7dda\u5f62\u306b\u5909\u63db\u3059\u308b\u3053\u3068\u3067\u3059</p>\u3002\n",
|
||||
"<p>Normalizing each feature to zero mean and unit variance could affect what the layer can represent. As an example paper illustrates that, if the inputs to a sigmoid are normalized most of it will be within <span translate=no>_^_0_^_</span> range where the sigmoid is linear. To overcome this each feature is scaled and shifted by two trained parameters <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span>. <span translate=no>_^_3_^_</span> where <span translate=no>_^_4_^_</span> is the output of the batch normalization layer.</p>\n": "<p>\u5404\u7279\u5fb4\u91cf\u3092\u5e73\u5747\u30bc\u30ed\u3068\u5358\u4f4d\u5206\u6563\u306b\u6b63\u898f\u5316\u3059\u308b\u3068\u3001\u30ec\u30a4\u30e4\u30fc\u304c\u8868\u73fe\u3067\u304d\u308b\u5185\u5bb9\u306b\u5f71\u97ff\u3059\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\u3002\u4f8b\u793a\u3057\u3066\u3044\u308b\u3088\u3046\u306b\u3001\u30b7\u30b0\u30e2\u30a4\u30c9\u3078\u306e\u5165\u529b\u304c\u6b63\u898f\u5316\u3055\u308c\u308b\u3068\u3001<span translate=no>_^_0_^_</span>\u305d\u306e\u307b\u3068\u3093\u3069\u306f\u30b7\u30b0\u30e2\u30a4\u30c9\u304c\u7dda\u5f62\u3067\u3042\u308b\u7bc4\u56f2\u5185\u306b\u306a\u308a\u307e\u3059\u3002\u3053\u308c\u3092\u89e3\u6c7a\u3059\u308b\u305f\u3081\u306b\u3001\u5404\u6a5f\u80fd\u306e\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3068\u30b7\u30d5\u30c8\u3092\u5b66\u7fd2\u6e08\u307f\u306e 2 <span translate=no>_^_1_^_</span> \u3064\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3068\u3067\u8abf\u6574\u3057\u307e\u3059\u3002<span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u3053\u3053\u3067<span translate=no>_^_4_^_</span>\u3001\u306f\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u5c64\u306e\u51fa\u529b\u3067\u3059</p>\u3002\n",
|
||||
"<p>Normalizing outside the gradient computation using pre-computed (detached) means and variances doesn't work. For instance. (ignoring variance), let <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> is a trained bias and <span translate=no>_^_3_^_</span> is an outside gradient computation (pre-computed constant).</p>\n": "<p>\u4e8b\u524d\u306b\u8a08\u7b97\u3055\u308c\u305f\uff08\u5206\u96e2\u3055\u308c\u305f\uff09\u5e73\u5747\u3068\u5206\u6563\u3092\u4f7f\u7528\u3057\u3066\u52fe\u914d\u8a08\u7b97\u306e\u5916\u3067\u6b63\u898f\u5316\u3059\u308b\u3053\u3068\u306f\u3067\u304d\u307e\u305b\u3093\u3002\u4f8b\u3048\u3070\u3002(\u5206\u6563\u306f\u7121\u8996)\u3001<span translate=no>_^_0_^_</span>\u3053\u3053\u3067\u3001<span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> \u306f\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u6e08\u307f\u306e\u30d0\u30a4\u30a2\u30b9\u3067\u3001\u5916\u90e8\u52fe\u914d\u8a08\u7b97 (\u4e8b\u524d\u306b\u8a08\u7b97\u3055\u308c\u305f\u5b9a\u6570) <span translate=no>_^_3_^_</span> \u3067\u3059</p>\u3002\n",
|
||||
"<p>Note that <span translate=no>_^_0_^_</span> has no effect on <span translate=no>_^_1_^_</span>. Therefore, <span translate=no>_^_2_^_</span> will increase or decrease based <span translate=no>_^_3_^_</span>, and keep on growing indefinitely in each training update. The paper notes that similar explosions happen with variances.</p>\n": "<p><span translate=no>_^_0_^_</span>\u306b\u306f\u5f71\u97ff\u3057\u306a\u3044\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044<span translate=no>_^_1_^_</span>\u3002\u3057\u305f\u304c\u3063\u3066\u3001<span translate=no>_^_2_^_</span>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u66f4\u65b0\u3059\u308b\u305f\u3073\u306b\u5897\u52a0\u307e\u305f\u306f\u6e1b\u5c11\u3057<span translate=no>_^_3_^_</span>\u3001\u7121\u671f\u9650\u306b\u6210\u9577\u3057\u7d9a\u3051\u307e\u3059\u3002\u3053\u306e\u8ad6\u6587\u306f\u3001\u540c\u69d8\u306e\u7206\u767a\u306b\u306f\u3070\u3089\u3064\u304d\u304c\u3042\u308b\u3068\u8ff0\u3079\u3066\u3044\u307e\u3059</p>\u3002\n",
|
||||
"<p>Note that when applying batch normalization after a linear transform like <span translate=no>_^_0_^_</span> the bias parameter <span translate=no>_^_1_^_</span> gets cancelled due to normalization. So you can and should omit bias parameter in linear transforms right before the batch normalization.</p>\n": "<p><span translate=no>_^_0_^_</span>\u7dda\u5f62\u5909\u63db\u306e\u3088\u3046\u306a\u7dda\u5f62\u5909\u63db\u306e\u5f8c\u306b\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092\u9069\u7528\u3059\u308b\u3068\u3001<span translate=no>_^_1_^_</span>\u6b63\u898f\u5316\u306b\u3088\u308a\u30d0\u30a4\u30a2\u30b9\u30d1\u30e9\u30e1\u30fc\u30bf\u304c\u30ad\u30e3\u30f3\u30bb\u30eb\u3055\u308c\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002\u305d\u306e\u305f\u3081\u3001\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306e\u76f4\u524d\u306b\u7dda\u5f62\u5909\u63db\u306e\u30d0\u30a4\u30a2\u30b9\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u7701\u7565\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u3001\u307e\u305f\u7701\u7565\u3059\u3079\u304d\u3067\u3059</p>\u3002\n",
|
||||
"<p>The paper defines <em>Internal Covariate Shift</em> as the change in the distribution of network activations due to the change in network parameters during training. For example, let's say there are two layers <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>. During the beginning of the training <span translate=no>_^_2_^_</span> outputs (inputs to <span translate=no>_^_3_^_</span>) could be in distribution <span translate=no>_^_4_^_</span>. Then, after some training steps, it could move to <span translate=no>_^_5_^_</span>. This is <em>internal covariate shift</em>.</p>\n": "<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<em>\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u3092</em>\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306e\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u306e\u5909\u5316\u306b\u3088\u308b\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u306e\u5206\u5e03\u306e\u5909\u5316\u3068\u3057\u3066\u5b9a\u7fa9\u3057\u3066\u3044\u307e\u3059\u3002\u305f\u3068\u3048\u3070\u3001<span translate=no>_^_0_^_</span>\u3068\u306e 2 \u3064\u306e\u30ec\u30a4\u30e4\u30fc\u304c\u3042\u308b\u3068\u3057\u307e\u3059<span translate=no>_^_1_^_</span>\u3002\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u958b\u59cb\u6642\u306b\u3001<span translate=no>_^_2_^_</span>\u30a2\u30a6\u30c8\u30d7\u30c3\u30c8\uff08\u3078\u306e\u30a4\u30f3\u30d7\u30c3\u30c8<span translate=no>_^_3_^_</span>\uff09\u304c\u914d\u5e03\u3055\u308c\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059<span translate=no>_^_4_^_</span>\u3002\u305d\u306e\u5f8c\u3001\u3044\u304f\u3064\u304b\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u624b\u9806\u3092\u5b9f\u884c\u3059\u308b\u3068\u3001\u306b\u79fb\u52d5\u3059\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059<span translate=no>_^_5_^_</span>\u3002<em>\u3053\u308c\u306f\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u3067\u3059</em></p>\u3002\n",
|
||||
"<p>The paper introduces a simplified version which they call <em>Batch Normalization</em>. First simplification is that it normalizes each feature independently to have zero mean and unit variance: <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span>-dimensional input.</p>\n": "<p>\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001<em>\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3068\u547c\u3070\u308c\u308b\u7c21\u7565\u7248\u3092\u7d39\u4ecb\u3057\u3066\u3044\u307e\u3059</em>\u30021 \u3064\u76ee\u306e\u7c21\u7565\u5316\u306f\u3001\u5404\u7279\u5fb4\u91cf\u3092\u72ec\u7acb\u3057\u3066\u5e73\u5747\u304c 0\u3001\u5358\u4f4d\u5206\u6563\u306b\u306a\u308b\u3088\u3046\u306b\u6b63\u898f\u5316\u3059\u308b\u3053\u3068\u3067\u3059\u3002<span translate=no>_^_0_^_</span>\u3053\u3053\u3067\u3001\u306f <span translate=no>_^_1_^_</span>-\u6b21\u5143\u306e\u5165\u529b\u3067\u3059</p>\u3002<span translate=no>_^_2_^_</span>\n",
|
||||
"<p>The second simplification is to use estimates of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> from the mini-batch for normalization; instead of calculating the mean and variance across the whole dataset.</p>\n": "<p>2 \u3064\u76ee\u306e\u7c21\u7565\u5316\u306f\u3001<span translate=no>_^_0_^_</span>\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u5168\u4f53\u306e\u5e73\u5747\u3068\u5206\u6563\u3092\u8a08\u7b97\u3059\u308b\u306e\u3067\u306f\u306a\u304f\u3001<span translate=no>_^_1_^_</span>\u30df\u30cb\u30d0\u30c3\u30c1\u304b\u3089\u306e\u5e73\u5747\u3068\u5206\u6563\u306e\u63a8\u5b9a\u5024\u3092\u6b63\u898f\u5316\u306b\u4f7f\u7528\u3059\u308b\u3053\u3068\u3067\u3059\u3002</p>\n",
|
||||
"<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch Normalization from paper <a href=\"https://arxiv.org/abs/1502.03167\">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a>.</p>\n": "<p>\u3053\u308c\u306f\u3001\u300c\u30d0\u30c3\u30c1\u6b63\u898f\u5316<a href=\"https://arxiv.org/abs/1502.03167\">:\u5185\u90e8\u5171\u5909\u91cf\u30b7\u30d5\u30c8\u3092\u6e1b\u3089\u3059\u3053\u3068\u306b\u3088\u308b\u30c7\u30a3\u30fc\u30d7\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u9ad8\u901f\u5316\u300d<a href=\"https://pytorch.org\">\u3068\u3044\u3046\u8ad6\u6587\u304b\u3089\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n",
|
||||
"<p>We need to know <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in order to perform the normalization. So during inference, you either need to go through the whole (or part of) dataset and find the mean and variance, or you can use an estimate calculated during training. The usual practice is to calculate an exponential moving average of mean and variance during the training phase and use that for inference.</p>\n": "<p>\u6b63\u898f\u5316\u3092\u5b9f\u884c\u3059\u308b\u306b\u306f<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u3001\u3068\u3092\u77e5\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\u305d\u306e\u305f\u3081\u3001\u63a8\u8ad6\u6642\u306b\u306f\u3001\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u5168\u4f53 (\u307e\u305f\u306f\u4e00\u90e8) \u3092\u8abf\u3079\u3066\u5e73\u5747\u3068\u5206\u6563\u3092\u6c42\u3081\u308b\u304b\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\u8a08\u7b97\u3055\u308c\u305f\u63a8\u5b9a\u5024\u3092\u4f7f\u7528\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\u901a\u5e38\u306f\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u6bb5\u968e\u3067\u5e73\u5747\u3068\u5206\u6563\u306e\u6307\u6570\u79fb\u52d5\u5e73\u5747\u3092\u8a08\u7b97\u3057\u3001\u305d\u308c\u3092\u63a8\u8ad6\u306b\u4f7f\u7528\u3057\u307e\u3059</p>\u3002\n",
|
||||
"<p>Whitening is computationally expensive because you need to de-correlate and the gradients must flow through the full whitening calculation.</p>\n": "<p>\u30db\u30ef\u30a4\u30c8\u30cb\u30f3\u30b0\u306f\u3001\u76f8\u95a2\u3092\u306a\u304f\u3059\u5fc5\u8981\u304c\u3042\u308a\u3001\u52fe\u914d\u304c\u30db\u30ef\u30a4\u30c8\u30cb\u30f3\u30b0\u306e\u8a08\u7b97\u5168\u4f53\u3092\u901a\u308b\u5fc5\u8981\u304c\u3042\u308b\u305f\u3081\u3001\u8a08\u7b97\u91cf\u304c\u591a\u304f\u306a\u308a\u307e\u3059\u3002</p>\n",
|
||||
"Batch Normalization": "\u30d0\u30c3\u30c1\u6b63\u898f\u5316"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">Batch Normalization</a></h1>\n": "<h1><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</a></h1>\n",
|
||||
"<h2>Inference</h2>\n": "<h2>\u0d85\u0db1\u0dd4\u0db8\u0dcf\u0db1\u0dba</h2>\n",
|
||||
"<h2>Normalization</h2>\n": "<h2>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u200d\u0dba\u0d9a\u0dbb\u0dab\u0dba</h2>\n",
|
||||
"<h3>Batch Normalization</h3>\n": "<h3>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</h3>\n",
|
||||
"<h3>Internal Covariate Shift</h3>\n": "<h3>\u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb\u0d9a\u0ddd\u0dc0\u0dbb\u0dd2\u0dba\u0db1\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0</h3>\n",
|
||||
"<h3>Normalizing outside gradient computation doesn't work</h3>\n": "<h3>\u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dd2\u0da7\u0dad \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0dba\u0dd2</h3>\n",
|
||||
"<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/011254fe647011ebbb8e0242ac1c0002\"><span translate=no>_^_1_^_</span></a> </p>\n": "<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/011254fe647011ebbb8e0242ac1c0002\"> <span translate=no>_^_1_^_</span></a> </p>\n",
|
||||
"<p>Batch normalization also makes the back propagation invariant to the scale of the weights and empirically it improves generalization, so it has regularization effects too.</p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db8\u0d9f\u0dd2\u0db1\u0dca \u0db4\u0dc3\u0dd4\u0db4\u0dc3 \u0db4\u0dca\u0dbb\u0da0\u0dcf\u0dbb\u0dab\u0dba \u0db6\u0dbb \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0da7 \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0d86\u0db1\u0dd4\u0db7\u0dc0\u0dd2\u0d9a \u0dbd\u0dd9\u0dc3 \u0d91\u0dba \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2, \u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d91\u0dba \u0dc0\u0dd2\u0db0\u0dd2\u0db8\u0dad\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db6\u0dbd\u0db4\u0dd1\u0db8\u0dca \u0daf \u0d87\u0dad. </p>\n",
|
||||
"<p>By stabilizing the distribution, batch normalization minimizes the internal covariate shift.</p>\n": "<p>\u0db6\u0dd9\u0daf\u0dcf\u0dc4\u0dd0\u0dbb\u0dd3\u0db8 \u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca, \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb \u0d9a\u0ddd\u0dc0\u0dd2\u0dbb\u0dda\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 \u0d85\u0dc0\u0db8 \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>Here's <a href=\"mnist.html\">the training code</a> and a notebook for training a CNN classifier that uses batch normalization for MNIST dataset.</p>\n": "<p>MNIST\u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc3\u0dd3\u0d91\u0db1\u0dca\u0d91\u0db1\u0dca \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 <a href=\"mnist.html\">\u0d9a\u0dda\u0dad\u0dba</a> \u0dc3\u0dc4 \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0db4\u0ddc\u0dad\u0d9a\u0dca \u0db8\u0dd9\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Internal covariate shift will adversely affect training speed because the later layers (<span translate=no>_^_0_^_</span> in the above example) have to adapt to this shifted distribution.</p>\n": "<p>\u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb\u0d9a\u0ddd\u0dc0\u0dd2\u0da0\u0dbb\u0dda\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dc0\u0dda\u0d9c\u0dba\u0da7 \u0d85\u0dc4\u0dd2\u0dad\u0d9a\u0dbb \u0dbd\u0dd9\u0dc3 \u0db6\u0dbd\u0db4\u0dcf\u0db1\u0dd4 \u0d87\u0dad, \u0db8\u0db1\u0dca\u0daf \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc3\u0dca\u0dae\u0dbb (\u0d89\u0dc4\u0dad<span translate=no>_^_0_^_</span> \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0dda) \u0db8\u0dd9\u0db8 \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dc5 \u0dc0\u0dca\u0dba\u0dcf\u0db4\u0dca\u0dad\u0dd2\u0dba\u0da7 \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0dc0\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4 \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dd2. </p>\n",
|
||||
"<p>It is known that whitening improves training speed and convergence. <em>Whitening</em> is linearly transforming inputs to have zero mean, unit variance, and be uncorrelated.</p>\n": "<p>\u0d91\u0dbawhitening \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dc0\u0dda\u0d9c\u0dba \u0dc4\u0dcf \u0d85\u0db7\u0dd2\u0dc3\u0dbb\u0dab\u0dba \u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1 \u0db6\u0dc0 \u0d9a\u0dc0\u0dd4\u0dbb\u0dd4\u0dad\u0dca \u0dc4\u0ddc\u0db3\u0dd2\u0db1\u0dca \u0daf\u0db1\u0dca\u0db1\u0dcf \u0d9a\u0dbb\u0dd4\u0dab\u0d9a\u0dd2. <em>\u0dc3\u0dd4\u0daf\u0dd4\u0d9a\u0dd2\u0dbb\u0dd3\u0db8</em> \u0dba\u0db1\u0dd4 \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba, \u0d92\u0d9a\u0d9a \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dc3\u0dc4 \u0dc3\u0dc4\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0 \u0db1\u0ddc\u0dc0\u0db1 \u0dbd\u0dd9\u0dc3 \u0dba\u0dd9\u0daf\u0dc0\u0dd4\u0db8\u0dca \u0dbb\u0dda\u0d9b\u0dd3\u0dba\u0dc0 \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2. </p>\n",
|
||||
"<p>Normalizing each feature to zero mean and unit variance could affect what the layer can represent. As an example paper illustrates that, if the inputs to a sigmoid are normalized most of it will be within <span translate=no>_^_0_^_</span> range where the sigmoid is linear. To overcome this each feature is scaled and shifted by two trained parameters <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span>. <span translate=no>_^_3_^_</span> where <span translate=no>_^_4_^_</span> is the output of the batch normalization layer.</p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d85\u0d82\u0d9c\u0dba \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba\u0da7 \u0dc4\u0dcf \u0d92\u0d9a\u0d9a \u0dc0\u0dd2\u0da0\u0dbd\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba\u0da7 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0db1\u0dd2\u0dba\u0ddd\u0da2\u0db1\u0dba \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0daf\u0dd9\u0dba\u0da7 \u0db6\u0dbd\u0db4\u0dcf\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0db6\u0dc0 \u0db4\u0dd9\u0db1\u0dca\u0db1\u0dd4\u0db8\u0dca, \u0d91\u0dba sigmoid \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0dba\u0dd9\u0daf\u0dc0\u0dd4\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db1\u0db8\u0dca, \u0db6\u0ddc\u0dc4\u0ddd \u0d91\u0dba \u0dc3\u0dd2\u0d9c\u0dca\u0db8\u0ddd\u0dba\u0dd2\u0da9\u0dca \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d9a\u0ddc\u0dc4\u0dd9\u0daf <span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dcf\u0dc3\u0dba \u0dad\u0dd4\u0dc5 \u0dc0\u0db1\u0dd4 \u0d87\u0dad. \u0db8\u0dd9\u0dba \u0da2\u0dba \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dd1\u0db8 \u0d85\u0d82\u0d9c\u0dba\u0d9a\u0dca\u0db8 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca <span translate=no>_^_1_^_</span> \u0daf\u0dd9\u0d9a\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0dd0\u0db6\u0dda <span translate=no>_^_2_^_</span>. <span translate=no>_^_3_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba <span translate=no>_^_4_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n",
|
||||
"<p>Normalizing outside the gradient computation using pre-computed (detached) means and variances doesn't work. For instance. (ignoring variance), let <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> is a trained bias and <span translate=no>_^_3_^_</span> is an outside gradient computation (pre-computed constant).</p>\n": "<p>\u0db4\u0dd6\u0dbb\u0dca\u0dc0\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf (\u0dc0\u0dd9\u0db1\u0dca\u0dc0\u0dd6) \u0db8\u0dcf\u0db0\u0dca\u0dba\u0dba\u0db1\u0dca \u0dc3\u0dc4 \u0dc0\u0dd2\u0da0\u0dbd\u0dca\u0dba\u0dba\u0db1\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd2\u0db1\u0dca \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda \u0d9c\u0dab\u0db1\u0dba \u0db4\u0dd2\u0da7\u0dad \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3. (\u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0db1\u0ddc\u0dc3\u0dbd\u0d9a\u0dcf \u0dc4\u0dd0\u0dbb\u0dd3\u0db8), \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0\u0d9a\u0dca <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf \u0dba\u0db1\u0dca\u0db1 \u0dc3\u0dc4 <span translate=no>_^_3_^_</span> \u0db4\u0dd2\u0da7\u0dad \u0dc1\u0dca\u0dbb\u0dda\u0dab\u0dd2\u0dba\u0dda \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dca (\u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db1\u0dd2\u0dba\u0dad\u0dba) \u0dc0\u0dda. <span translate=no>_^_2_^_</span> </p>\n",
|
||||
"<p>Note that <span translate=no>_^_0_^_</span> has no effect on <span translate=no>_^_1_^_</span>. Therefore, <span translate=no>_^_2_^_</span> will increase or decrease based <span translate=no>_^_3_^_</span>, and keep on growing indefinitely in each training update. The paper notes that similar explosions happen with variances.</p>\n": "<p>\u0d9a\u0dd2\u0dc3\u0dd2\u0daf\u0dd4\u0db6\u0dbd\u0db4\u0dd1\u0db8\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0d87\u0dad\u0dd2 \u0db1\u0ddc\u0dc0\u0db1 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span>. \u0d92 \u0db1\u0dd2\u0dc3\u0dcf, \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd0\u0da9\u0dd2 \u0dc4\u0ddd \u0d85\u0da9\u0dd4 <span translate=no>_^_2_^_</span> \u0dc0\u0db1\u0dd4 \u0d87\u0dad <span translate=no>_^_3_^_</span>, \u0dc3\u0dc4 \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0daf\u0dd2\u0db1 \u0db1\u0dd2\u0dba\u0db8\u0dba\u0d9a\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2\u0dc0 \u0dc0\u0dbb\u0dca\u0db0\u0db1\u0dba \u0daf\u0dd2\u0d9c\u0da7\u0db8. \u0dc3\u0db8\u0dcf\u0db1 \u0db4\u0dd2\u0db4\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0dc0\u0dd9\u0db1\u0dc3\u0dca\u0d9a\u0db8\u0dca \u0dc3\u0db8\u0d9f \u0dc3\u0dd2\u0daf\u0dd4\u0dc0\u0db1 \u0db6\u0dc0 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc3\u0da7\u0dc4\u0db1\u0dca \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>Note that when applying batch normalization after a linear transform like <span translate=no>_^_0_^_</span> the bias parameter <span translate=no>_^_1_^_</span> gets cancelled due to normalization. So you can and should omit bias parameter in linear transforms right before the batch normalization.</p>\n": "<p>\u0dbb\u0dda\u0d9b\u0dd3\u0dba\u0db4\u0dbb\u0dd2\u0dab\u0dcf\u0db8\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dba\u0ddc\u0daf\u0db1 \u0dc0\u0dd2\u0da7 \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba <span translate=no>_^_1_^_</span> \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc4\u0dda\u0dad\u0dd4\u0dc0\u0dd9\u0db1\u0dca \u0d85\u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0dc0\u0db1 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1. <span translate=no>_^_0_^_</span> \u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0da7 \u0db4\u0dd9\u0dbb \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0dda \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd2\u0dba \u0d94\u0db6\u0da7 \u0db8\u0d9f \u0dc4\u0dd0\u0dbb\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4\u0dba. </p>\n",
|
||||
"<p>The paper defines <em>Internal Covariate Shift</em> as the change in the distribution of network activations due to the change in network parameters during training. For example, let's say there are two layers <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>. During the beginning of the training <span translate=no>_^_2_^_</span> outputs (inputs to <span translate=no>_^_3_^_</span>) could be in distribution <span translate=no>_^_4_^_</span>. Then, after some training steps, it could move to <span translate=no>_^_5_^_</span>. This is <em>internal covariate shift</em>.</p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0da2\u0dcf\u0dbd \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0dc0\u0dd3\u0db8 \u0dc4\u0dda\u0dad\u0dd4\u0dc0\u0dd9\u0db1\u0dca \u0da2\u0dcf\u0dbd \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0db6\u0dd9\u0daf\u0dcf \u0dc4\u0dd0\u0dbb\u0dd3\u0db8\u0dda \u0dc0\u0dd9\u0db1\u0dc3 \u0dbd\u0dd9\u0dc3 <em>\u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb \u0d9a\u0ddd\u0dc0\u0dbb\u0dd3\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0</em> \u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3, \u0dc3\u0dca\u0dae\u0dbb \u0daf\u0dd9\u0d9a\u0d9a\u0dca \u0d87\u0dad\u0dd2 \u0db6\u0dc0 <span translate=no>_^_0_^_</span> \u0d9a\u0dd2\u0dba\u0db8\u0dd4 <span translate=no>_^_1_^_</span>. \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7\u0dba\u0dda \u0daf\u0dd3 ( <span translate=no>_^_2_^_</span> \u0dba\u0dd9\u0daf\u0dc0\u0dd4\u0db8\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 <span translate=no>_^_3_^_</span>) \u0db6\u0dd9\u0daf\u0dcf\u0dc4\u0dd0\u0dbb\u0dd3\u0db8\u0dda \u0dc0\u0dd2\u0dba \u0dc4\u0dd0\u0d9a\u0dd2 <span translate=no>_^_4_^_</span>. \u0dc3\u0db8\u0dc4\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0db4\u0dc3\u0dd4\u0dc0, \u0d91\u0dba \u0dc0\u0dd9\u0dad \u0d9c\u0db8\u0db1\u0dca \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba <span translate=no>_^_5_^_</span>. \u0db8\u0dd9\u0dba <em>\u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb \u0d9a\u0ddd\u0dc0\u0dd2\u0dbb\u0dda\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0\u0d9a\u0dd2</em>. </p>\n",
|
||||
"<p>The paper introduces a simplified version which they call <em>Batch Normalization</em>. First simplification is that it normalizes each feature independently to have zero mean and unit variance: <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span>-dimensional input.</p>\n": "<p>\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2\u0dc3\u0dbb\u0dbd \u0d9a\u0dc5 \u0d85\u0db1\u0dd4\u0dc0\u0dcf\u0daf\u0dba\u0d9a\u0dca \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0d85\u0dad\u0dbb \u0d91\u0dba <em>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</em>\u0dbd\u0dd9\u0dc3 \u0dc4\u0dd0\u0db3\u0dd2\u0db1\u0dca\u0dc0\u0dda. \u0db4\u0dc5\u0db8\u0dd4 \u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db1\u0db8\u0dca, \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0dc3\u0dc4 \u0d92\u0d9a\u0d9a \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dad\u0dd2\u0db6\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d85\u0d82\u0d9c\u0dba \u0dc3\u0dca\u0dc0\u0dcf\u0db0\u0dd3\u0db1\u0dc0 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2: <span translate=no>_^_2_^_</span>-dimensional <span translate=no>_^_1_^_</span> \u0d86\u0daf\u0dcf\u0db1\u0dba <span translate=no>_^_0_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n",
|
||||
"<p>The second simplification is to use estimates of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> from the mini-batch for normalization; instead of calculating the mean and variance across the whole dataset.</p>\n": "<p>\u0daf\u0dd9\u0dc0\u0db1\u0dc3\u0dbb\u0dbd \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db1\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dd4\u0da9\u0dcf <span translate=no>_^_1_^_</span> \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dd9\u0db1\u0dca \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba <span translate=no>_^_0_^_</span> \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2; \u0db8\u0dd4\u0dc5\u0dd4 \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db4\u0dd4\u0dbb\u0dcf\u0db8 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7. </p>\n",
|
||||
"<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch Normalization from paper <a href=\"https://arxiv.org/abs/1502.03167\">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a>.</p>\n": "<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc0\u0dd9\u0dad\u0dd2\u0db1\u0dca <a href=\"https://arxiv.org/abs/1502.03167\">\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8: \u0d85\u0db7\u0dca\u0dba\u0db1\u0dca\u0dad\u0dbb \u0d9a\u0ddd\u0dc0\u0dbb\u0dd2\u0dba\u0da7\u0dca \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 \u0d85\u0da9\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0d9c\u0dd0\u0db9\u0dd4\u0dbb\u0dd4 \u0da2\u0dcf\u0dbd \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca</a> \u0d9a\u0dd2\u0dbb\u0dd3\u0db8. </p>\n",
|
||||
"<p>We need to know <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in order to perform the normalization. So during inference, you either need to go through the whole (or part of) dataset and find the mean and variance, or you can use an estimate calculated during training. The usual practice is to calculate an exponential moving average of mean and variance during the training phase and use that for inference.</p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4 \u0daf\u0dd0\u0db1\u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4 <span translate=no>_^_0_^_</span> \u0d85\u0dad\u0dbb <span translate=no>_^_1_^_</span> \u0d91\u0dba \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0dba. \u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d85\u0db1\u0dd4\u0db8\u0dcf\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3, \u0d94\u0db6\u0da7 \u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab (\u0dc4\u0ddd \u0d9a\u0ddc\u0da7\u0dc3\u0d9a\u0dca) \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0d9c\u0ddc\u0dc3\u0dca \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dc3\u0ddc\u0dba\u0dcf \u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4\u0dba, \u0db1\u0dd0\u0dad\u0dc4\u0ddc\u0dad\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. \u0dc3\u0dd4\u0db4\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc0\u0db1\u0dca\u0db1\u0dda \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d85\u0dc0\u0db0\u0dd2\u0dba\u0dda\u0daf\u0dd3 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba \u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0\u0dba\u0dda on \u0dcf\u0dad\u0dd3\u0dba \u0da0\u0dbd\u0db1\u0dba \u0dc0\u0db1 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0d85\u0db1\u0dd4\u0db8\u0dcf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2. </p>\n",
|
||||
"<p>Whitening is computationally expensive because you need to de-correlate and the gradients must flow through the full whitening calculation.</p>\n": "<p>\u0d94\u0db6\u0daf-\u0dc3\u0dc4\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0db1\u0dd2\u0dc3\u0dcf whitening \u0db4\u0dbb\u0dd2\u0d9c\u0dab\u0d9a\u0db8\u0dba \u0db8\u0dd2\u0dbd \u0d85\u0db0\u0dd2\u0d9a \u0dc0\u0db1 \u0d85\u0dad\u0dbb, \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab whitening \u0d9c\u0dab\u0db1\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0d9c\u0dbd\u0dcf \u0dba\u0dd4\u0dad\u0dd4\u0dba. </p>\n",
|
||||
"Batch Normalization": "\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">Batch Normalization</a></h1>\n": "<h1><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">\u6279\u91cf\u6807\u51c6\u5316</a></h1>\n",
|
||||
"<h2>Inference</h2>\n": "<h2>\u63a8\u65ad</h2>\n",
|
||||
"<h2>Normalization</h2>\n": "<h2>\u89c4\u8303\u5316</h2>\n",
|
||||
"<h3>Batch Normalization</h3>\n": "<h3>\u6279\u91cf\u6807\u51c6\u5316</h3>\n",
|
||||
"<h3>Internal Covariate Shift</h3>\n": "<h3>\u5185\u90e8\u534f\u53d8\u91cf\u79fb\u4f4d</h3>\n",
|
||||
"<h3>Normalizing outside gradient computation doesn't work</h3>\n": "<h3>\u5bf9\u5916\u90e8\u68af\u5ea6\u8ba1\u7b97\u8fdb\u884c\u5f52\u4e00\u5316\u4e0d\u8d77\u4f5c\u7528</h3>\n",
|
||||
"<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a> </p>\n": "<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<p>Batch normalization also makes the back propagation invariant to the scale of the weights and empirically it improves generalization, so it has regularization effects too.</p>\n": "<p>\u6279\u91cf\u5f52\u4e00\u5316\u8fd8\u4f7f\u53cd\u5411\u4f20\u64ad\u4e0e\u6743\u91cd\u7684\u6bd4\u4f8b\u4fdd\u6301\u4e0d\u53d8\uff0c\u4ece\u7ecf\u9a8c\u4e0a\u8bb2\uff0c\u5b83\u6539\u5584\u4e86\u6cdb\u5316\uff0c\u56e0\u6b64\u5b83\u4e5f\u5177\u6709\u6b63\u5219\u5316\u6548\u679c\u3002</p>\n",
|
||||
"<p>By stabilizing the distribution, batch normalization minimizes the internal covariate shift.</p>\n": "<p>\u901a\u8fc7\u7a33\u5b9a\u5206\u5e03\uff0c\u6279\u91cf\u5f52\u4e00\u5316\u53ef\u4ee5\u6700\u5927\u9650\u5ea6\u5730\u51cf\u5c11\u5185\u90e8\u534f\u53d8\u91cf\u504f\u79fb\u3002</p>\n",
|
||||
"<p>Here's <a href=\"mnist.html\">the training code</a> and a notebook for training a CNN classifier that uses batch normalization for MNIST dataset.</p>\n": "<p><a href=\"mnist.html\">\u4ee5\u4e0b\u662f\u8bad\u7ec3\u4ee3\u7801</a>\u548c\u7528\u4e8e\u8bad\u7ec3 CNN \u5206\u7c7b\u5668\u7684\u7b14\u8bb0\u672c\uff0c\u8be5\u5206\u7c7b\u5668\u4f7f\u7528 MNIST \u6570\u636e\u96c6\u7684\u6279\u91cf\u5f52\u4e00\u5316\u3002</p>\n",
|
||||
"<p>Internal covariate shift will adversely affect training speed because the later layers (<span translate=no>_^_0_^_</span> in the above example) have to adapt to this shifted distribution.</p>\n": "<p>\u5185\u90e8\u534f\u53d8\u91cf\u504f\u79fb\u5c06\u5bf9\u8bad\u7ec3\u901f\u5ea6\u4ea7\u751f\u4e0d\u5229\u5f71\u54cd\uff0c\u56e0\u4e3a\u540e\u9762\u7684\u56fe\u5c42\uff08\u5728\u4e0a\u9762\u7684\u4f8b\u5b50<span translate=no>_^_0_^_</span>\u4e2d\uff09\u5fc5\u987b\u9002\u5e94\u8fd9\u79cd\u504f\u79fb\u5206\u5e03\u3002</p>\n",
|
||||
"<p>It is known that whitening improves training speed and convergence. <em>Whitening</em> is linearly transforming inputs to have zero mean, unit variance, and be uncorrelated.</p>\n": "<p>\u4f17\u6240\u5468\u77e5\uff0c\u7f8e\u767d\u53ef\u4ee5\u63d0\u9ad8\u8bad\u7ec3\u901f\u5ea6\u548c\u6536\u655b\u6027\u3002<em>\u7f8e\u767d</em>\u662f\u5c06\u8f93\u5165\u8fdb\u884c\u7ebf\u6027\u53d8\u6362\uff0c\u4f7f\u5176\u5747\u503c\u4e3a\u96f6\u3001\u5355\u4f4d\u65b9\u5dee\u4e14\u4e0d\u76f8\u5173\u3002</p>\n",
|
||||
"<p>Normalizing each feature to zero mean and unit variance could affect what the layer can represent. As an example paper illustrates that, if the inputs to a sigmoid are normalized most of it will be within <span translate=no>_^_0_^_</span> range where the sigmoid is linear. To overcome this each feature is scaled and shifted by two trained parameters <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span>. <span translate=no>_^_3_^_</span> where <span translate=no>_^_4_^_</span> is the output of the batch normalization layer.</p>\n": "<p>\u5c06\u6bcf\u4e2a\u8981\u7d20\u5f52\u4e00\u5316\u4e3a\u96f6\u5747\u503c\u548c\u5355\u4f4d\u65b9\u5dee\u53ef\u80fd\u4f1a\u5f71\u54cd\u56fe\u5c42\u53ef\u4ee5\u8868\u793a\u7684\u5185\u5bb9\u3002\u4f5c\u4e3a\u793a\u4f8b\u8bba\u6587\u8bf4\u660e\uff0c\u5982\u679csigmoid\u7684\u8f93\u5165\u88ab\u5f52\u4e00\u5316\uff0c\u5219\u5927\u90e8\u5206\u5c06\u5728sigmoid\u4e3a\u7ebf\u6027\u7684<span translate=no>_^_0_^_</span>\u8303\u56f4\u5185\u3002\u4e3a\u4e86\u514b\u670d\u8fd9\u4e2a\u95ee\u9898\uff0c\u6bcf\u4e2a\u7279\u5f81\u90fd\u901a\u8fc7\u4e24\u4e2a\u7ecf\u8fc7\u8bad\u7ec3\u7684\u53c2\u6570\u8fdb\u884c\u7f29\u653e<span translate=no>_^_1_^_</span>\u548c\u79fb\u52a8<span translate=no>_^_2_^_</span>\u3002<span translate=no>_^_3_^_</span>\u5176\u4e2d<span translate=no>_^_4_^_</span>\u662f\u6279\u91cf\u5f52\u4e00\u5316\u5c42\u7684\u8f93\u51fa\u3002</p>\n",
|
||||
"<p>Normalizing outside the gradient computation using pre-computed (detached) means and variances doesn't work. For instance. (ignoring variance), let <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> is a trained bias and <span translate=no>_^_3_^_</span> is an outside gradient computation (pre-computed constant).</p>\n": "<p>\u4f7f\u7528\u9884\u5148\u8ba1\u7b97\uff08\u5206\u79bb\uff09\u5747\u503c\u548c\u65b9\u5dee\u5728\u68af\u5ea6\u8ba1\u7b97\u4e4b\u5916\u8fdb\u884c\u5f52\u4e00\u5316\u4e0d\u8d77\u4f5c\u7528\u3002\u4f8b\u5982\u3002\uff08\u5ffd\u7565\u65b9\u5dee\uff09<span translate=no>_^_0_^_</span>\uff0c\u8ba9 wher<span translate=no>_^_1_^_</span> e an<span translate=no>_^_2_^_</span> d \u662f\u4e00\u4e2a\u8bad\u7ec3\u8fc7\u7684\u504f\u5dee\uff0c<span translate=no>_^_3_^_</span>\u662f\u5916\u90e8\u68af\u5ea6\u8ba1\u7b97\uff08\u9884\u5148\u8ba1\u7b97\u7684\u5e38\u91cf\uff09\u3002</p>\n",
|
||||
"<p>Note that <span translate=no>_^_0_^_</span> has no effect on <span translate=no>_^_1_^_</span>. Therefore, <span translate=no>_^_2_^_</span> will increase or decrease based <span translate=no>_^_3_^_</span>, and keep on growing indefinitely in each training update. The paper notes that similar explosions happen with variances.</p>\n": "<p>\u8bf7\u6ce8\u610f<span translate=no>_^_0_^_</span>\uff0c\u8fd9\u5bf9<span translate=no>_^_1_^_</span>\u3002\u56e0\u6b64\uff0c<span translate=no>_^_2_^_</span>\u5c06\u5728\u6bcf\u6b21\u8bad\u7ec3\u66f4\u65b0\u4e2d\u589e\u52a0\u6216\u51cf\u5c11<span translate=no>_^_3_^_</span>\uff0c\u5e76\u4e14\u4f1a\u65e0\u9650\u671f\u5730\u589e\u957f\u3002\u8be5\u62a5\u6307\u51fa\uff0c\u7c7b\u4f3c\u7684\u7206\u70b8\u4f1a\u53d1\u751f\u5dee\u5f02\u3002</p>\n",
|
||||
"<p>Note that when applying batch normalization after a linear transform like <span translate=no>_^_0_^_</span> the bias parameter <span translate=no>_^_1_^_</span> gets cancelled due to normalization. So you can and should omit bias parameter in linear transforms right before the batch normalization.</p>\n": "<p>\u8bf7\u6ce8\u610f\uff0c\u5728\u7ebf\u6027\u53d8\u6362\u4e4b\u540e\u5e94\u7528\u6279\u91cf\u5f52\u4e00\u5316\u65f6\uff0c\u6bd4\u5982<span translate=no>_^_0_^_</span>\u504f\u7f6e\u53c2\u6570<span translate=no>_^_1_^_</span>\u4f1a\u56e0\u5f52\u4e00\u5316\u800c\u88ab\u53d6\u6d88\u3002\u56e0\u6b64\uff0c\u4f60\u53ef\u4ee5\u800c\u4e14\u5e94\u8be5\u5728\u6279\u91cf\u5f52\u4e00\u5316\u4e4b\u524d\u7701\u7565\u7ebf\u6027\u53d8\u6362\u4e2d\u7684\u504f\u7f6e\u53c2\u6570\u3002</p>\n",
|
||||
"<p>The paper defines <em>Internal Covariate Shift</em> as the change in the distribution of network activations due to the change in network parameters during training. For example, let's say there are two layers <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>. During the beginning of the training <span translate=no>_^_2_^_</span> outputs (inputs to <span translate=no>_^_3_^_</span>) could be in distribution <span translate=no>_^_4_^_</span>. Then, after some training steps, it could move to <span translate=no>_^_5_^_</span>. This is <em>internal covariate shift</em>.</p>\n": "<p>\u672c\u6587\u5c06<em>\u5185\u90e8\u534f\u53d8\u91cf\u79fb\u4f4d</em>\u5b9a\u4e49\u4e3a\u8bad\u7ec3\u671f\u95f4\u7531\u4e8e\u7f51\u7edc\u53c2\u6570\u7684\u53d8\u5316\u800c\u5bfc\u81f4\u7684\u7f51\u7edc\u6fc0\u6d3b\u5206\u5e03\u7684\u53d8\u5316\u3002\u4f8b\u5982\uff0c\u5047\u8bbe\u6709\u4e24\u5c42<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span>\u3002\u5728\u57f9\u8bad\u5f00\u59cb\u65f6\uff0c\u53ef\u4ee5\u5206\u53d1<span translate=no>_^_2_^_</span>\u8f93\u51fa\uff08\u8f93\u5165<span translate=no>_^_3_^_</span>\uff09<span translate=no>_^_4_^_</span>\u3002\u7136\u540e\uff0c\u7ecf\u8fc7\u4e00\u4e9b\u8bad\u7ec3\u6b65\u9aa4\u540e\uff0c\u5b83\u53ef\u80fd\u4f1a\u79fb\u81f3<span translate=no>_^_5_^_</span>\u3002\u8fd9\u662f<em>\u5185\u90e8\u534f\u53d8\u91cf\u79fb\u4f4d</em>\u3002</p>\n",
|
||||
"<p>The paper introduces a simplified version which they call <em>Batch Normalization</em>. First simplification is that it normalizes each feature independently to have zero mean and unit variance: <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span>-dimensional input.</p>\n": "<p>\u672c\u6587\u4ecb\u7ecd\u4e86\u4e00\u4e2a\u7b80\u5316\u7248\u672c\uff0c\u4ed6\u4eec\u79f0\u4e4b\u4e3a<em>\u6279\u91cf\u89c4\u8303\u5316</em>\u3002\u9996\u5148\u7b80\u5316\u7684\u662f\uff0c\u5b83\u5c06\u6bcf\u4e2a\u8981\u7d20\u72ec\u7acb\u5f52\u4e00\u5316\uff0c\u4f7f\u5176\u5747\u503c\u548c\u5355\u4f4d\u65b9\u5dee\u4e3a\u96f6\uff1a<span translate=no>_^_0_^_</span>\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f<span translate=no>_^_2_^_</span>\u7ef4\u5ea6\u8f93\u5165\u3002</p>\n",
|
||||
"<p>The second simplification is to use estimates of mean <span translate=no>_^_0_^_</span> and variance <span translate=no>_^_1_^_</span> from the mini-batch for normalization; instead of calculating the mean and variance across the whole dataset.</p>\n": "<p>\u7b2c\u4e8c\u79cd\u7b80\u5316\u65b9\u6cd5\u662f\u4f7f\u7528<span translate=no>_^_1_^_</span>\u6765\u81ea\u5fae\u578b\u6279\u6b21\u7684\u5747\u503c<span translate=no>_^_0_^_</span>\u548c\u65b9\u5dee\u7684\u4f30\u8ba1\u503c\u8fdb\u884c\u5f52\u4e00\u5316\uff1b\u800c\u4e0d\u662f\u8ba1\u7b97\u6574\u4e2a\u6570\u636e\u96c6\u7684\u5747\u503c\u548c\u65b9\u5dee\u3002</p>\n",
|
||||
"<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Batch Normalization from paper <a href=\"https://arxiv.org/abs/1502.03167\">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a>.</p>\n": "<p>\u8fd9\u662f <a href=\"https://pytorch.org\">PyTorch</a> \u4ece\u7eb8\u8d28\u6279\u91cf\u89c4\u8303\u5316\u4e2d<a href=\"https://arxiv.org/abs/1502.03167\">\u5b9e\u73b0\u6279\u91cf\u5f52\u4e00\u5316\uff1a\u901a\u8fc7\u51cf\u5c11\u5185\u90e8\u534f\u53d8\u91cf\u504f\u79fb\u52a0\u901f\u6df1\u5ea6\u7f51\u7edc\u8bad\u7ec3</a>\u3002</p>\n",
|
||||
"<p>We need to know <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> in order to perform the normalization. So during inference, you either need to go through the whole (or part of) dataset and find the mean and variance, or you can use an estimate calculated during training. The usual practice is to calculate an exponential moving average of mean and variance during the training phase and use that for inference.</p>\n": "<p>\u6211\u4eec\u9700\u8981\u77e5\u9053 an<span translate=no>_^_0_^_</span> d<span translate=no>_^_1_^_</span> \u624d\u80fd\u6267\u884c\u89c4\u8303\u5316\u3002\u56e0\u6b64\uff0c\u5728\u63a8\u7406\u8fc7\u7a0b\u4e2d\uff0c\u60a8\u8981\u4e48\u9700\u8981\u904d\u5386\u6574\u4e2a\uff08\u6216\u90e8\u5206\uff09\u6570\u636e\u96c6\u5e76\u627e\u5230\u5747\u503c\u548c\u65b9\u5dee\uff0c\u8981\u4e48\u53ef\u4ee5\u4f7f\u7528\u8bad\u7ec3\u671f\u95f4\u8ba1\u7b97\u7684\u4f30\u8ba1\u503c\u3002\u901a\u5e38\u7684\u505a\u6cd5\u662f\u5728\u8bad\u7ec3\u9636\u6bb5\u8ba1\u7b97\u5747\u503c\u548c\u65b9\u5dee\u7684\u6307\u6570\u79fb\u52a8\u5e73\u5747\u7ebf\uff0c\u7136\u540e\u5c06\u5176\u7528\u4e8e\u63a8\u65ad\u3002</p>\n",
|
||||
"<p>Whitening is computationally expensive because you need to de-correlate and the gradients must flow through the full whitening calculation.</p>\n": "<p>\u7f8e\u767d\u5728\u8ba1\u7b97\u4e0a\u5f88\u6602\u8d35\uff0c\u56e0\u4e3a\u4f60\u9700\u8981\u53bb\u5173\u8054\uff0c\u800c\u4e14\u68af\u5ea6\u5fc5\u987b\u901a\u8fc7\u5b8c\u6574\u7684\u7f8e\u767d\u8ba1\u7b97\u3002</p>\n",
|
||||
"Batch Normalization": "\u6279\u91cf\u6807\u51c6\u5316"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">DeepNorm</a> Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">\u30c7\u30a3\u30fc\u30d7\u30fb\u30ce\u30fc\u30e0\u30fb\u30a8\u30af\u30b9\u30da\u30ea\u30e1\u30f3\u30c8</a></h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n<p>This is a autoregressive transformer model that uses DeepNorm.</p>\n": "<h2>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n<p>\u3053\u308c\u306fDeepNorm\u3092\u4f7f\u7528\u3059\u308b\u81ea\u5df1\u56de\u5e30\u5909\u63db\u30e2\u30c7\u30eb\u3067\u3059\u3002</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u3053\u308c\u306f\u4ee5\u4e0b\u304b\u3089\u7d99\u627f\u3055\u308c\u307e\u3059 <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h4>Calculate <span translate=no>_^_0_^_</span></h4>\n<p><span translate=no>_^_1_^_</span></p>\n": "<h4>\u8a08\u7b97 <span translate=no>_^_0_^_</span></h4>\n<p><span translate=no>_^_1_^_</span></p>\n",
|
||||
"<h4>Create and run the experiment</h4>\n": "<h4>\u5b9f\u9a13\u3092\u4f5c\u6210\u3057\u3066\u5b9f\u884c\u3059\u308b</h4>\n",
|
||||
"<h4>Initialize the model</h4>\n": "<h4>\u30e2\u30c7\u30eb\u3092\u521d\u671f\u5316</h4>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for DeepNorm </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u305d\u3057\u3066\u30c7\u30a3\u30fc\u30d7\u30ce\u30fc\u30e0\u7528</p>\n",
|
||||
"<p>Adam optimizer with no warmup </p>\n": "<p>\u30a6\u30a9\u30fc\u30e0\u30a2\u30c3\u30d7\u306a\u3057\u306e Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the token embeddings </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u5165\u624b</p>\n",
|
||||
"<p>Model </p>\n": "<p>\u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>Number of heads in the attention </p>\n": "<p>\u6ce8\u76ee\u3055\u308c\u3066\u3044\u308b\u30d8\u30c3\u30c9\u306e\u6570</p>\n",
|
||||
"<p>Number of layers </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u6570</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Readout layer </p>\n": "<p>\u8aad\u307f\u51fa\u3057\u5c64</p>\n",
|
||||
"<p>Return results </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3059</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set model(s) for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3057\u307e\u3059</p>\n",
|
||||
"<p>Size of each attention head </p>\n": "<p>\u5404\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Token embedding layer </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32 \u30a8\u30dd\u30c3\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Transformer with <span translate=no>_^_0_^_</span> layers </p>\n": "<p><span translate=no>_^_0_^_</span>\u5c64\u4ed8\u304d\u5909\u5727\u5668</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are the input tokens of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u5f62\u72b6\u306e\u5165\u529b\u30c8\u30fc\u30af\u30f3\u3067\u3059 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of tokens in the vocabulary </li>\n<li><span translate=no>_^_1_^_</span> is the embedding size </li>\n<li><span translate=no>_^_2_^_</span> is the number of transformer layers </li>\n<li><span translate=no>_^_3_^_</span> is the layer. We use <span translate=no>_^_4_^_</span> copies of this for the tranformer.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u5185\u306e\u30c8\u30fc\u30af\u30f3\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</li>\n<li><span translate=no>_^_2_^_</span>\u5909\u5727\u5668\u5c64\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_3_^_</span>\u30ec\u30a4\u30e4\u30fc\u3067\u3059\u3002<span translate=no>_^_4_^_</span>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u306b\u306f\u3053\u308c\u306e\u30b3\u30d4\u30fc\u3092\u4f7f\u3044\u307e\u3059</li></ul>\u3002\n",
|
||||
"DeepNorm Experiment": "\u30c7\u30a3\u30fc\u30d7\u30fb\u30ce\u30fc\u30e0\u30fb\u30a8\u30af\u30b9\u30da\u30ea\u30e1\u30f3\u30c8",
|
||||
"Training a DeepNorm transformer on Tiny Shakespeare.": "\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u3067\u30c7\u30a3\u30fc\u30d7\u30ce\u30fc\u30e0\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u3092\u8a13\u7df4\u3057\u3066\u308b"
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">DeepNorm</a> Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">\u0d9c\u0dd0\u0db9\u0dd4\u0dbb\u0dd4 \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba</a> \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n<p>This is a autoregressive transformer model that uses DeepNorm.</p>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n<p>\u0db8\u0dd9\u0dbaDeepNorm \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dd2. </p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0db1\u0dca\u0db1\u0dda <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h4>Calculate <span translate=no>_^_0_^_</span></h4>\n<p><span translate=no>_^_1_^_</span></p>\n": "<h4>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span></h4>\n<p><span translate=no>_^_1_^_</span></p>\n",
|
||||
"<h4>Create and run the experiment</h4>\n": "<h4>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1</h4>\n",
|
||||
"<h4>Initialize the model</h4>\n": "<h4>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h4>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for DeepNorm </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc3\u0dc4 \u0d9c\u0dd0\u0db9\u0dd4\u0dbb\u0dd4 \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba <span translate=no>_^_1_^_</span> \u0dc3\u0db3\u0dc4\u0dcf </p>\n",
|
||||
"<p>Adam optimizer with no warmup </p>\n": "<p>\u0d8b\u0db1\u0dd4\u0dc3\u0dd4\u0db8\u0dca\u0dc0\u0dd3\u0db8\u0d9a\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the token embeddings </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>Number of heads in the attention </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1 \u0dc4\u0dd2\u0dc3\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of layers </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dbb\u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Readout layer </p>\n": "<p>\u0d9a\u0dd2\u0dba\u0dc0\u0dd3\u0db8\u0dda\u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Return results </p>\n": "<p>\u0d86\u0db4\u0dc3\u0dd4\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db5\u0dbd </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set model(s) for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba (\u0dba) \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Size of each attention head </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc4\u0dd2\u0dc3 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Token embedding layer </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32\u0dc0\u0dba\u0dc3 \u0d85\u0dc0\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Transformer with <span translate=no>_^_0_^_</span> layers </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc3\u0dca\u0dae\u0dbb \u0dc3\u0dc4\u0dd2\u0dad \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are the input tokens of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d86\u0daf\u0dcf\u0db1 \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dda <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of tokens in the vocabulary </li>\n<li><span translate=no>_^_1_^_</span> is the embedding size </li>\n<li><span translate=no>_^_2_^_</span> is the number of transformer layers </li>\n<li><span translate=no>_^_3_^_</span> is the layer. We use <span translate=no>_^_4_^_</span> copies of this for the tranformer.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0\u0dda \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_1_^_</span> \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dc0\u0dda </li>\n<li><span translate=no>_^_2_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0dca\u0dae\u0dbb \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_3_^_</span> \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0dc0\u0dda. \u0d85\u0db4\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dd9\u0db4\u0ddd\u0db8\u0dbb\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dd9\u0dc4\u0dd2 <span translate=no>_^_4_^_</span> \u0db4\u0dd2\u0da7\u0db4\u0dad\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4. </li></ul>\n",
|
||||
"DeepNorm Experiment": "\u0d9c\u0dd0\u0db9\u0dd4\u0dbb\u0dd4 \u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"Training a DeepNorm transformer on Tiny Shakespeare.": "\u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0db8\u0dad \u0da9\u0dd3\u0db4\u0dca \u0db1\u0ddd\u0db8\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8."
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">DeepNorm</a> Experiment</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">\u6df1\u5ea6\u89c4\u8303</a>\u5b9e\u9a8c</h1>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/deep_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Auto-Regressive model</h2>\n<p>This is a autoregressive transformer model that uses DeepNorm.</p>\n": "<h2>\u81ea\u56de\u5f52\u6a21\u578b</h2>\n<p>\u8fd9\u662f\u4e00\u4e2a\u4f7f\u7528 DeepNorm \u7684\u81ea\u56de\u5f52\u53d8\u538b\u5668\u6a21\u578b\u3002</p>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea <a href=\"../../experiments/nlp_autoregression.html#NLPAutoRegressionConfigs\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h4>Calculate <span translate=no>_^_0_^_</span></h4>\n<p><span translate=no>_^_1_^_</span></p>\n": "<h4>\u8ba1\u7b97<span translate=no>_^_0_^_</span></h4>\n<p><span translate=no>_^_1_^_</span></p>\n",
|
||||
"<h4>Create and run the experiment</h4>\n": "<h4>\u521b\u5efa\u5e76\u8fd0\u884c\u5b9e\u9a8c</h4>\n",
|
||||
"<h4>Initialize the model</h4>\n": "<h4>\u521d\u59cb\u5316\u6a21\u578b</h4>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for DeepNorm </p>\n": "<p><span translate=no>_^_0_^_</span>\u5bf9<span translate=no>_^_1_^_</span>\u4e8e DeepNorm</p>\n",
|
||||
"<p>Adam optimizer with no warmup </p>\n": "<p>\u6ca1\u6709\u9884\u70ed\u7684 Adam \u4f18\u5316\u5668</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Get logits </p>\n": "<p>\u83b7\u53d6\u65e5\u5fd7</p>\n",
|
||||
"<p>Get the token embeddings </p>\n": "<p>\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Model </p>\n": "<p>\u578b\u53f7</p>\n",
|
||||
"<p>Number of heads in the attention </p>\n": "<p>\u5173\u6ce8\u7684\u5934\u90e8\u6570\u91cf</p>\n",
|
||||
"<p>Number of layers </p>\n": "<p>\u5c42\u6570</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Readout layer </p>\n": "<p>\u8bfb\u51fa\u5c42</p>\n",
|
||||
"<p>Return results </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set model(s) for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Size of each attention head </p>\n": "<p>\u6bcf\u4e2a\u6ce8\u610f\u5934\u7684\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>Token embedding layer </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u5c42</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>\u8bad\u7ec3 32 \u4e2a\u65f6\u4ee3</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
|
||||
"<p>Transformer with <span translate=no>_^_0_^_</span> layers </p>\n": "<p>\u5e26<span translate=no>_^_0_^_</span>\u5c42\u7684\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are the input tokens of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u7684\u8f93\u5165\u6807\u8bb0<span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of tokens in the vocabulary </li>\n<li><span translate=no>_^_1_^_</span> is the embedding size </li>\n<li><span translate=no>_^_2_^_</span> is the number of transformer layers </li>\n<li><span translate=no>_^_3_^_</span> is the layer. We use <span translate=no>_^_4_^_</span> copies of this for the tranformer.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8bcd\u6c47\u8868\u4e2d\u4ee3\u5e01\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5d4c\u5165\u7684\u5927\u5c0f</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u53d8\u538b\u5668\u5c42\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_3_^_</span>\u662f\u5c42\u3002\u6211\u4eec\u5728\u53d8\u538b\u5668\u4e0a\u4f7f\u7528\u8fd9\u4e2a<span translate=no>_^_4_^_</span>\u526f\u672c\u3002</li></ul>\n",
|
||||
"DeepNorm Experiment": "\u6df1\u5ea6\u89c4\u8303\u5b9e\u9a8c",
|
||||
"Training a DeepNorm transformer on Tiny Shakespeare.": "\u5728\u5c0f\u838e\u58eb\u6bd4\u4e9a\u4e0a\u8bad\u7ec3 DeepNorm \u53d8\u538b\u5668\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Group Normalization</h1>\n": "<h1>\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u306e\u305f\u3081\u306e CIFAR10 \u5b9f\u9a13</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u30e2\u30c7\u30eb\u4f5c\u6210</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n": "<h3>CIFAR-10 \u5206\u985e\u7528\u306e VGG \u30e2\u30c7\u30eb</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Convolution, Normalization and Activation layers </p>\n": "<p>\u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u3001\u30ce\u30fc\u30de\u30e9\u30a4\u30bc\u30fc\u30b7\u30e7\u30f3\u3001\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Create a sequential model with the layers </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u3092\u542b\u3080\u30b7\u30fc\u30b1\u30f3\u30b7\u30e3\u30eb\u30e2\u30c7\u30eb\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u69cb\u6210\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Final linear layer </p>\n": "<p>\u6700\u7d42\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Final logits layer </p>\n": "<p>\u6700\u7d42\u30ed\u30b8\u30c3\u30c8\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Max pooling at end of each block </p>\n": "<p>\u5404\u30d6\u30ed\u30c3\u30af\u7d42\u4e86\u6642\u306e\u6700\u5927\u30d7\u30fc\u30ea\u30f3\u30b0</p>\n",
|
||||
"<p>Number of channels in each layer in each block </p>\n": "<p>\u5404\u30d6\u30ed\u30c3\u30af\u306e\u5404\u30ec\u30a4\u30e4\u30fc\u306e\u30c1\u30e3\u30f3\u30cd\u30eb\u6570</p>\n",
|
||||
"<p>Number of groups </p>\n": "<p>\u30b0\u30eb\u30fc\u30d7\u6570</p>\n",
|
||||
"<p>RGB channels </p>\n": "<p>RGB \u30c1\u30e3\u30f3\u30cd\u30eb</p>\n",
|
||||
"<p>Reshape for classification layer </p>\n": "<p>\u5206\u985e\u30ec\u30a4\u30e4\u30fc\u306e\u5f62\u72b6\u3092\u5909\u66f4</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3092\u5b9f\u884c\u3057\u307e\u3059</p>\n",
|
||||
"<p>The VGG layers </p>\n": "<p>VGG \u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"CIFAR10 Experiment to try Group Normalization": "\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u3092\u8a66\u3059\u305f\u3081\u306e CIFAR10 \u5b9f\u9a13",
|
||||
"This trains is a simple convolutional neural network that uses group normalization to classify CIFAR10 images.": "\u3053\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306f\u3001\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3057\u3066 CIFAR10 \u753b\u50cf\u3092\u5206\u985e\u3059\u308b\u5358\u7d14\u306a\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Group Normalization</h1>\n": "<h1>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf CIFAR10 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n": "<h3>CIFA-10\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf VGG \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Convolution, Normalization and Activation layers </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db8\u0dd4\u0dad\u0dd2\u0dba, \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dc4 \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0dca\u0dae\u0dbb </p>\n",
|
||||
"<p>Create a sequential model with the layers </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dbb\u0dc3\u0db8\u0d9f \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Final linear layer </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Final logits layer </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0db1\u0dca\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Max pooling at end of each block </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d9a\u0ddc\u0da7\u0dc3 \u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0dba\u0dda \u0db8\u0dd0\u0d9a\u0dca\u0dc3\u0dca \u0dad\u0da7\u0dcf\u0d9a </p>\n",
|
||||
"<p>Number of channels in each layer in each block </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d9a\u0ddc\u0da7\u0dc3\u0dd9\u0dc4\u0dd2 \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of groups </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>RGB channels </p>\n": "<p>RGB\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf </p>\n",
|
||||
"<p>Reshape for classification layer </p>\n": "<p>\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dc3\u0dca\u0dae\u0dbb\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0dc3\u0d9a\u0dc3\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>The VGG layers </p>\n": "<p>VGG\u0dc3\u0dca\u0dae\u0dbb </p>\n",
|
||||
"CIFAR10 Experiment to try Group Normalization": "\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d8b\u0dad\u0dca\u0dc3\u0dcf\u0dc4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf CIFAR10 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This trains is a simple convolutional neural network that uses group normalization to classify CIFAR10 images.": "\u0db8\u0dd9\u0db8 \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba CIFAR10 \u0dbb\u0dd6\u0db4 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc3\u0dbb\u0dbd \u0dc3\u0d82\u0dc0\u0dc4\u0db1 \u0dc3\u0dca\u0db1\u0dcf\u0dba\u0dd4\u0d9a \u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Group Normalization</h1>\n": "<h1>CIFAR10 \u7fa4\u5f52\u4e00\u5316\u5b9e\u9a8c</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u521b\u5efa\u6a21\u578b</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Number of groups </p>\n": "<p>\u7ec4\u6570</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c\u5e76\u8fd0\u884c\u8bad\u7ec3\u5faa\u73af</p>\n",
|
||||
"CIFAR10 Experiment to try Group Normalization": "CIFAR10 \u5c1d\u8bd5\u7fa4\u5f52\u4e00\u5316\u7684\u5b9e\u9a8c",
|
||||
"This trains is a simple convolutional neural network that uses group normalization to classify CIFAR10 images.": "\u8be5\u5217\u8f66\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff0c\u5b83\u4f7f\u7528\u7fa4\u5f52\u4e00\u5316\u5bf9 CIFAR10 \u56fe\u50cf\u8fdb\u884c\u5206\u7c7b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/group_norm/index.html\">Group Normalization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the <a href=\"https://arxiv.org/abs/1803.08494\">Group Normalization</a> paper.</p>\n<p><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">Batch Normalization</a> works well for large enough batch sizes but not well for small batch sizes, because it normalizes over the batch. Training large models with large batch sizes is not possible due to the memory capacity of the devices.</p>\n<p>This paper introduces Group Normalization, which normalizes a set of features together as a group. This is based on the observation that classical features such as <a href=\"https://en.wikipedia.org/wiki/Scale-invariant_feature_transform\">SIFT</a> and <a href=\"https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients\">HOG</a> are group-wise features. The paper proposes dividing feature channels into groups and then separately normalizing all channels within each group.</p>\n<p>Here's a <a href=\"https://nn.labml.ai/normalization/group_norm/experiment.html\">CIFAR 10 classification model</a> that uses instance normalization.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/group_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/group_norm/index.html\">\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316</a></h1>\n<p><a href=\"https://pytorch.org\"><a href=\"https://arxiv.org/abs/1803.08494\">\u3053\u308c\u306f\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u8ad6\u6587\u306ePyTorch\u5b9f\u88c5\u3067\u3059</a></a>\u3002</p>\n<p><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306f\u30d0\u30c3\u30c1\u5168\u4f53\u3067\u6b63\u898f\u5316\u3055\u308c\u308b\u305f\u3081</a>\u3001\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u304c\u5341\u5206\u5927\u304d\u3044\u5834\u5408\u306f\u3046\u307e\u304f\u6a5f\u80fd\u3057\u307e\u3059\u304c\u3001\u5c0f\u3055\u306a\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u306b\u306f\u9069\u3057\u3066\u3044\u307e\u305b\u3093\u3002\u30c7\u30d0\u30a4\u30b9\u306e\u30e1\u30e2\u30ea\u5bb9\u91cf\u306b\u3088\u308a\u3001\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u306e\u5927\u304d\u3044\u5927\u898f\u6a21\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306f\u4e0d\u53ef\u80fd\u3067\u3059\u3002</p>\n<p>\u672c\u7a3f\u3067\u306f\u3001\u4e00\u9023\u306e\u7279\u5fb4\u3092\u30b0\u30eb\u30fc\u30d7\u3068\u3057\u3066\u307e\u3068\u3081\u3066\u6b63\u898f\u5316\u3059\u308b\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316\u306b\u3064\u3044\u3066\u7d39\u4ecb\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u3001<a href=\"https://en.wikipedia.org/wiki/Scale-invariant_feature_transform\"><a href=\"https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients\">SIFT\u3084HOG\u306a\u3069\u306e\u53e4\u5178\u7684\u7279\u5fb4\u306f\u30b0\u30eb\u30fc\u30d7\u3054\u3068\u306e\u7279\u5fb4\u3067\u3042\u308b\u3068\u3044\u3046\u89b3\u5bdf\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a></a>\u3002\u3053\u306e\u8ad6\u6587\u3067\u306f\u3001\u30d5\u30a3\u30fc\u30c1\u30e3\u30c1\u30e3\u30cd\u30eb\u3092\u30b0\u30eb\u30fc\u30d7\u306b\u5206\u5272\u3057\u3001\u5404\u30b0\u30eb\u30fc\u30d7\u5185\u306e\u3059\u3079\u3066\u306e\u30c1\u30e3\u30cd\u30eb\u3092\u500b\u5225\u306b\u6b63\u898f\u5316\u3059\u308b\u3053\u3068\u3092\u63d0\u6848\u3057\u3066\u3044\u307e\u3059</p>\u3002\n<p>\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306e\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3059\u308b <a href=\"https://nn.labml.ai/normalization/group_norm/experiment.html\">CIFAR 10 \u5206\u985e\u30e2\u30c7\u30eb\u3092\u6b21\u306b\u793a\u3057\u307e\u3059</a>\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/group_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"Group Normalization": "\u30b0\u30eb\u30fc\u30d7\u6b63\u898f\u5316"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/group_norm/index.html\">Group Normalization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the <a href=\"https://arxiv.org/abs/1803.08494\">Group Normalization</a> paper.</p>\n<p><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">Batch Normalization</a> works well for large enough batch sizes but not well for small batch sizes, because it normalizes over the batch. Training large models with large batch sizes is not possible due to the memory capacity of the devices.</p>\n<p>This paper introduces Group Normalization, which normalizes a set of features together as a group. This is based on the observation that classical features such as <a href=\"https://en.wikipedia.org/wiki/Scale-invariant_feature_transform\">SIFT</a> and <a href=\"https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients\">HOG</a> are group-wise features. The paper proposes dividing feature channels into groups and then separately normalizing all channels within each group.</p>\n<p>Here's a <a href=\"https://nn.labml.ai/normalization/group_norm/experiment.html\">CIFAR 10 classification model</a> that uses group normalization.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/group_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/group_norm/index.html\">\u7fa4\u7ec4\u6807\u51c6\u5316</a></h1>\n<p>\u8fd9\u662f <a href=\"https://pytorch.org\">PyTorch</a> \u5bf9<a href=\"https://arxiv.org/abs/1803.08494\">\u7fa4\u7ec4\u6807\u51c6\u5316</a>\u8bba\u6587\u7684\u5b9e\u73b0\u3002</p>\n<p><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">\u6279\u91cf\u6807\u51c6\u5316</a>\u9002\u7528\u4e8e\u8db3\u591f\u5927\u7684\u6279\u91cf\u5927\u5c0f\uff0c\u4f46\u5bf9\u4e8e\u5c0f\u6279\u91cf\u6765\u8bf4\u5374\u4e0d\u592a\u597d\uff0c\u56e0\u4e3a\u5b83\u4f1a\u5728\u6279\u6b21\u4e0a\u8fdb\u884c\u6807\u51c6\u5316\u3002\u7531\u4e8e\u8bbe\u5907\u7684\u5185\u5b58\u5bb9\u91cf\uff0c\u65e0\u6cd5\u8bad\u7ec3\u6279\u91cf\u8f83\u5927\u7684\u5927\u578b\u6a21\u578b\u3002</p>\n<p>\u672c\u6587\u4ecb\u7ecd\u4e86\u7fa4\u7ec4\u5f52\u4e00\u5316\uff0c\u5b83\u5c06\u4e00\u7ec4\u7279\u5f81\u5f52\u4e00\u5316\u4e3a\u4e00\u4e2a\u7ec4\u3002\u8fd9\u662f\u57fa\u4e8e\u8fd9\u6837\u7684\u89c2\u5bdf\uff0c\u5373\u8bf8\u5982 <a href=\"https://en.wikipedia.org/wiki/Scale-invariant_feature_transform\">SIFT</a> \u548c <a href=\"https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients\">HO</a> G\u4e4b\u7c7b\u7684\u7ecf\u5178\u7279\u5f81\u662f\u6309\u7ec4\u5212\u5206\u7684\u7279\u5f81\u3002\u8be5\u8bba\u6587\u5efa\u8bae\u5c06\u7279\u5f81\u4fe1\u9053\u5206\u6210\u7ec4\uff0c\u7136\u540e\u5206\u522b\u5bf9\u6bcf\u4e2a\u7ec4\u5185\u7684\u6240\u6709\u4fe1\u9053\u8fdb\u884c\u6807\u51c6\u5316\u3002</p>\n<p>\u8fd9\u662f\u4f7f\u7528\u5b9e\u4f8b\u6807\u51c6\u5316\u7684 <a href=\"https://nn.labml.ai/normalization/group_norm/experiment.html\">CIFAR 10 \u5206\u7c7b\u6a21\u578b</a>\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/group_norm/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"Group Normalization": "\u7fa4\u7ec4\u89c4\u8303\u5316"
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>Instance Normalization</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.08022\">Instance Normalization: The Missing Ingredient for Fast Stylization</a>.</p>\n<p>Instance normalization was introduced to improve <a href=\"https://paperswithcode.com/task/style-transfer\">style transfer</a>. It is based on the observation that stylization should not depend on the contrast of the content image. The "contrast normalization" is</p>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a batch of images with dimensions image index <span translate=no>_^_2_^_</span>, feature channel <span translate=no>_^_3_^_</span>, and spatial position <span translate=no>_^_4_^_</span>.</p>\n<p>Since it's hard for a convolutional network to learn "contrast normalization", this paper introduces instance normalization which does that.</p>\n<p>Here's a <a href=\"experiment.html\">CIFAR 10 classification model</a> that uses instance normalization.</p>\n": "<h1>\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316</h1>\n<p>\u3053\u308c\u306f <a href=\"https://pytorch.org\">PyTorch</a> \u306e\u300c<a href=\"https://arxiv.org/abs/1607.08022\">\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316:\u9ad8\u901f\u30b9\u30bf\u30a4\u30eb\u5316\u306b\u6b20\u3051\u3066\u3044\u308b\u8981\u7d20</a>\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n<p><a href=\"https://paperswithcode.com/task/style-transfer\">\u30b9\u30bf\u30a4\u30eb\u8ee2\u9001\u3092\u6539\u5584\u3059\u308b\u305f\u3081\u306b\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306e\u6b63\u898f\u5316\u304c\u5c0e\u5165\u3055\u308c\u307e\u3057\u305f</a>\u3002\u3053\u308c\u306f\u3001\u30b9\u30bf\u30a4\u30eb\u8a2d\u5b9a\u306f\u30b3\u30f3\u30c6\u30f3\u30c4\u753b\u50cf\u306e\u30b3\u30f3\u30c8\u30e9\u30b9\u30c8\u306b\u4f9d\u5b58\u3059\u3079\u304d\u3067\u306f\u306a\u3044\u3068\u3044\u3046\u89b3\u5bdf\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059\u3002\u300c\u30b3\u30f3\u30c8\u30e9\u30b9\u30c8\u6b63\u898f\u5316\u300d\u3068\u306f</p>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> \u306f\u3001\u753b\u50cf\u30a4\u30f3\u30c7\u30c3\u30af\u30b9<span translate=no>_^_2_^_</span>\u3001\u30d5\u30a3\u30fc\u30c1\u30e3\u30c1\u30e3\u30cd\u30eb<span translate=no>_^_3_^_</span>\u3001<span translate=no>_^_4_^_</span>\u304a\u3088\u3073\u7a7a\u9593\u4f4d\u7f6e\u3092\u542b\u3080\u753b\u50cf\u306e\u30d0\u30c3\u30c1\u3067\u3059\u3002</p>\n<p>\u7573\u307f\u8fbc\u307f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u306f\u300c\u30b3\u30f3\u30c8\u30e9\u30b9\u30c8\u6b63\u898f\u5316\u300d\u3092\u5b66\u7fd2\u3059\u308b\u306e\u306f\u96e3\u3057\u3044\u306e\u3067\u3001\u672c\u7a3f\u3067\u306f\u305d\u308c\u3092\u884c\u3046\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002</p>\n<p>\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306e\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3059\u308b <a href=\"experiment.html\">CIFAR 10 \u5206\u985e\u30e2\u30c7\u30eb\u3092\u6b21\u306b\u793a\u3057\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h2>Instance Normalization Layer</h2>\n<p>Instance normalization layer <span translate=no>_^_0_^_</span> normalizes the input <span translate=no>_^_1_^_</span> as follows:</p>\n<p>When input <span translate=no>_^_2_^_</span> is a batch of image representations, where <span translate=no>_^_3_^_</span> is the batch size, <span translate=no>_^_4_^_</span> is the number of channels, <span translate=no>_^_5_^_</span> is the height and <span translate=no>_^_6_^_</span> is the width. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span>. The affine transformation with <span translate=no>_^_9_^_</span> and <span translate=no>_^_10_^_</span> are optional.</p>\n<p><span translate=no>_^_11_^_</span></p>\n": "<h2>\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc</h2>\n<p>\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc\u306f\u3001<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u6b21\u306e\u3088\u3046\u306b\u5165\u529b\u3092\u6b63\u898f\u5316\u3057\u307e\u3059\u3002</p>\n<p><span translate=no>_^_2_^_</span>\u5165\u529b\u304c\u30a4\u30e1\u30fc\u30b8\u8868\u73fe\u306e\u30d0\u30c3\u30c1\u306e\u5834\u5408\u3001<span translate=no>_^_3_^_</span>\u306f\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3001<span translate=no>_^_4_^_</span>\u306f\u30c1\u30e3\u30cd\u30eb\u6570\u3001<span translate=no>_^_5_^_</span>\u306f\u9ad8\u3055\u3001<span translate=no>_^_6_^_</span>\u306f\u5e45\u3067\u3059\u3002<span translate=no>_^_7_^_</span>\u3068<span translate=no>_^_8_^_</span>\u3002<span translate=no>_^_9_^_</span><span translate=no>_^_10_^_</span>\u304a\u3088\u3073\u3092\u4f7f\u7528\u3057\u305f\u30a2\u30d5\u30a3\u30f3\u5909\u63db\u306f\u30aa\u30d7\u30b7\u30e7\u30f3\u3067\u3059</p>\u3002\n<p><span translate=no>_^_11_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5f62\u72b6\u306e\u30c6\u30f3\u30bd\u30eb\u3067\u3059\u3002<span translate=no>_^_2_^_</span>\u4efb\u610f\u306e\u6570 (0 \u306e\u5834\u5408\u3082\u3042\u308a\u307e\u3059) \u306e\u6b21\u5143\u3092\u793a\u3057\u307e\u3059\u3002\u305f\u3068\u3048\u3070\u3001\u753b\u50cf (2D) \u306e\u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u3067\u306f\u3001\u6b21\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059</p>\u3002<span translate=no>_^_3_^_</span>\n",
|
||||
"<p> Simple test</p>\n": "<p>\u7c21\u5358\u306a\u30c6\u30b9\u30c8</p>\n",
|
||||
"<p>Calculate the mean across last dimension i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u5f8c\u306e\u6b21\u5143\u306e\u5e73\u5747\u3001\u3064\u307e\u308a\u5404\u7279\u5fb4\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u521d\u3068\u6700\u5f8c\u306e\u6b21\u5143\u306e\u4e8c\u4e57\u5e73\u5747\u3001\u3064\u307e\u308a\u5404\u7279\u5fb4\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create parameters for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for scale and shift </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u30b9\u30b1\u30fc\u30eb\u3068\u30b7\u30d5\u30c8\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u3068\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u5143\u306e\u5f62\u3092\u4fdd\u3064</p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30ce\u30fc\u30de\u30e9\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5f62\u3092\u5909\u3048\u3066 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u5143\u306e\u5f62\u306b\u623b\u3057\u3066\u623b\u3059</p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u6a5f\u80fd\u306e\u6570\u304c\u540c\u3058\u3067\u3042\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3059\u308b\u305f\u3081\u306e\u30b5\u30cb\u30c6\u30a3\u30c1\u30a7\u30c3\u30af</p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30b1\u30fc\u30eb\u3068\u30b7\u30d5\u30c8 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5404\u6a5f\u80fd\u306e\u5dee\u7570 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is whether to scale and shift the normalized value</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u5165\u529b\u5185\u306e\u7279\u5fb4\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span><span translate=no>_^_3_^_</span>\u6570\u5024\u306e\u5b89\u5b9a\u6027\u306e\u305f\u3081\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u6b63\u898f\u5316\u3055\u308c\u305f\u5024\u3092\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3057\u3066\u30b7\u30d5\u30c8\u3059\u308b\u304b\u3069\u3046\u304b\u3067\u3059</li></ul>\n",
|
||||
"A PyTorch implementation/tutorial of instance normalization.": "PyTorch\u306e\u5b9f\u88c5/\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316\u306e\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"Instance Normalization": "\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316"
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>Instance Normalization</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.08022\">Instance Normalization: The Missing Ingredient for Fast Stylization</a>.</p>\n<p>Instance normalization was introduced to improve <a href=\"https://paperswithcode.com/task/style-transfer\">style transfer</a>. It is based on the observation that stylization should not depend on the contrast of the content image. The "contrast normalization" is</p>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a batch of images with dimensions image index <span translate=no>_^_2_^_</span>, feature channel <span translate=no>_^_3_^_</span>, and spatial position <span translate=no>_^_4_^_</span>.</p>\n<p>Since it's hard for a convolutional network to learn "contrast normalization", this paper introduces instance normalization which does that.</p>\n<p>Here's a <a href=\"experiment.html\">CIFAR 10 classification model</a> that uses instance normalization.</p>\n": "<h1>\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/1607.08022\">\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba: \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0dc1\u0ddb\u0dbd\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0dad\u0dd4\u0dbb\u0dd4\u0daf\u0dc4\u0db1\u0dca \u0dc0\u0dd6 \u0d85\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dc0\u0dca\u0dba\u0dba</a> . </p>\n<p><a href=\"https://paperswithcode.com/task/style-transfer\">\u0dc1\u0ddb\u0dbd\u0dd2\u0dba \u0db8\u0dcf\u0dbb\u0dd4</a>\u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0dbd\u0daf\u0dd3. \u0d91\u0dba \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda \u0d85\u0db1\u0dca\u0dad\u0dbb\u0dca\u0d9c\u0dad \u0dbb\u0dd6\u0db4\u0dba\u0dda \u0dc0\u0dd9\u0db1\u0dc3 \u0db8\u0dad \u0dc1\u0ddb\u0dbd\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0db3\u0dcf \u0db1\u0ddc\u0dad\u0dd2\u0db6\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4 \u0db6\u0dc0 \u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db8\u0dad \u0dba. \u201c\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc0\u0dd2\u0dbb\u0dd4\u0daf\u0dca\u0db0 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u201d \u0dc0\u0dda</p>\n<p><span translate=no>_^_0_^_</span></p>\n<p>\u0db8\u0dcf\u0db1\u0dbb\u0dd6\u0db4 \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba <span translate=no>_^_2_^_</span>, \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf\u0dc0 <span translate=no>_^_3_^_</span>\u0dc3\u0dc4 <span translate=no>_^_4_^_</span>\u0d85\u0dc0\u0d9a\u0dcf\u0dc1\u0dd3\u0dba \u0db4\u0dd2\u0dc4\u0dd2\u0da7\u0dd3\u0db8 \u0dc3\u0dc4\u0dd2\u0dad \u0dbb\u0dd6\u0db4 \u0dc3\u0db8\u0dd6\u0dc4\u0dba\u0d9a\u0dca <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dc4\u0dda\u0daf? </p>\n<p>\u0dc3\u0db8\u0dca\u0db8\u0dd4\u0dad\u0dd2\u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0da7 \u201c\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc0\u0dd2\u0dbb\u0dd4\u0daf\u0dca\u0db0 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u201d \u0d89\u0d9c\u0dd9\u0db1 \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0daf\u0dd4\u0dc2\u0dca\u0d9a\u0dbb \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca, \u0db8\u0dd9\u0db8 \u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0db8\u0d9f\u0dd2\u0db1\u0dca \u0d91\u0dba \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0db1 \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0dba\u0dd2. </p>\n<p>\u0db8\u0dd9\u0db1\u0dca\u0db1 <a href=\"experiment.html\">CIFAR \u0dad\u0dd2\u0dba\u0dd9\u0db1\u0dca\u0db1\u0dda 10 \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</a> . </p>\n",
|
||||
"<h2>Instance Normalization Layer</h2>\n<p>Instance normalization layer <span translate=no>_^_0_^_</span> normalizes the input <span translate=no>_^_1_^_</span> as follows:</p>\n<p>When input <span translate=no>_^_2_^_</span> is a batch of image representations, where <span translate=no>_^_3_^_</span> is the batch size, <span translate=no>_^_4_^_</span> is the number of channels, <span translate=no>_^_5_^_</span> is the height and <span translate=no>_^_6_^_</span> is the width. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span>. The affine transformation with <span translate=no>_^_9_^_</span> and <span translate=no>_^_10_^_</span> are optional.</p>\n<p><span translate=no>_^_11_^_</span></p>\n": "<h2>\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba</h2>\n<p>\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba \u0db4\u0dc4\u0dad \u0dc3\u0db3\u0dc4\u0db1\u0dca <span translate=no>_^_1_^_</span> \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0d86\u0daf\u0dcf\u0db1 <span translate=no>_^_0_^_</span> normalizes:</p>\n<p><span translate=no>_^_2_^_</span> \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dbb\u0dd6\u0db4 \u0db1\u0dd2\u0dbb\u0dd6\u0db4\u0dab \u0dc3\u0db8\u0dd6\u0dc4\u0dba\u0d9a\u0dca <span translate=no>_^_3_^_</span> \u0dc0\u0db1 \u0dc0\u0dd2\u0da7, \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf, <span translate=no>_^_4_^_</span> \u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0d9c\u0dab\u0db1, <span translate=no>_^_5_^_</span> \u0d8b\u0dc3 \u0dc3\u0dc4 <span translate=no>_^_6_^_</span> \u0db4\u0dc5\u0dbd \u0dc0\u0dda. <span translate=no>_^_7_^_</span> \u0dc3\u0dc4 <span translate=no>_^_8_^_</span>. \u0dc3\u0db8\u0d9c affine \u0db4\u0dbb\u0dd2\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba <span translate=no>_^_9_^_</span> \u0dc4\u0dcf \u0dc0\u0dd2\u0d9a\u0dbd\u0dca\u0db4 <span translate=no>_^_10_^_</span> \u0dc0\u0dda. </p>\n<p><span translate=no>_^_11_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p> <span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d86\u0dad\u0db1\u0dca\u0dba <span translate=no>_^_1_^_</span>\u0dc0\u0dda. <span translate=no>_^_2_^_</span> \u0d95\u0db1\u0dd1\u0db8 \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0\u0d9a\u0dca (\u0dc3\u0db8\u0dc4\u0dbb\u0dc0\u0dd2\u0da7 0) \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0daf\u0d9a\u0dca\u0dc0\u0dba\u0dd2. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3, \u0dbb\u0dd6\u0db4\u0dba\u0d9a\u0dca (2D) \u0dc3\u0d82\u0d9a\u0ddd\u0da0\u0db1\u0dba \u0dad\u0dd4\u0dc5 \u0db8\u0dd9\u0dba \u0dc0\u0db1\u0dd4 \u0d87\u0dad <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p> Simple test</p>\n": "<p> \u0dc3\u0dbb\u0dbd\u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba</p>\n",
|
||||
"<p>Calculate the mean across last dimension i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0db8\u0dcf\u0db1\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1, \u0d91\u0db1\u0db8\u0dca \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0dbd\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dcf\u0db0\u0dca\u0dba\u0dba\u0db1\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0dc4\u0dcf \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dc4\u0dbb\u0dc4\u0dcf \u0dc0\u0dbb\u0dca\u0d9c \u0d9a\u0dc5 \u0db8\u0db0\u0dca\u0dba\u0db1\u0dca\u0dba\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1; \u0d91\u0db1\u0db8\u0dca \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0dbd\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dcf\u0db0\u0dca\u0dba\u0dba\u0db1\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create parameters for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for scale and shift </p>\n": "<p>\u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0dc3\u0dc4 \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 <span translate=no>_^_1_^_</span> \u0dc3\u0db3\u0dc4\u0dcf <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u0db8\u0dd4\u0dbd\u0dca\u0dc4\u0dd0\u0da9\u0dba \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u200d\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db1\u0dd0\u0dc0\u0dad\u0dc4\u0dd0\u0da9\u0d9c\u0dc3\u0dca\u0dc0\u0dcf \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u0db8\u0dd4\u0dbd\u0dca\u0db4\u0dd2\u0da7\u0db4\u0dad\u0da7 \u0db1\u0dd0\u0dc0\u0dad \u0dc4\u0dd0\u0da9\u0d9c\u0dc3\u0dca\u0dc0\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0db4\u0dd0\u0db8\u0dd2\u0dab\u0dd3\u0db8 </p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c\u0d9c\u0dab\u0db1 \u0dc3\u0db8\u0dcf\u0db1 \u0db6\u0dc0 \u0dad\u0dc4\u0dc0\u0dd4\u0dbb\u0dd4 \u0d9a\u0dbb \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0db1\u0dd3\u0db4\u0dcf\u0dbb\u0d9a\u0dca\u0dc2\u0dcf\u0dc0 \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba\u0dc3\u0dc4 \u0db8\u0dcf\u0dbb\u0dd4\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dbd\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is whether to scale and shift the normalized value</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0d86\u0daf\u0dcf\u0db1\u0dba\u0dda \u0d87\u0dad\u0dd2 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_1_^_</span> \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc3\u0dca\u0dae\u0dcf\u0dba\u0dd2\u0dad\u0dcf\u0dc0 <span translate=no>_^_3_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0dc0\u0dda <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_4_^_</span> \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dc5 \u0d85\u0d9c\u0dba \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1\u0dba\u0dd2</li></ul>\n",
|
||||
"A PyTorch implementation/tutorial of instance normalization.": "\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dbd\u0dd9\u0dc3 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba.",
|
||||
"Instance Normalization": "\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"<h1>Instance Normalization</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.08022\">Instance Normalization: The Missing Ingredient for Fast Stylization</a>.</p>\n<p>Instance normalization was introduced to improve <a href=\"https://paperswithcode.com/task/style-transfer\">style transfer</a>. It is based on the observation that stylization should not depend on the contrast of the content image. The "contrast normalization" is</p>\n<p><span translate=no>_^_0_^_</span></p>\n<p>where <span translate=no>_^_1_^_</span> is a batch of images with dimensions image index <span translate=no>_^_2_^_</span>, feature channel <span translate=no>_^_3_^_</span>, and spatial position <span translate=no>_^_4_^_</span>.</p>\n<p>Since it's hard for a convolutional network to learn "contrast normalization", this paper introduces instance normalization which does that.</p>\n<p>Here's a <a href=\"experiment.html\">CIFAR 10 classification model</a> that uses instance normalization.</p>\n": "<h1>\u5b9e\u4f8b\u89c4\u8303\u5316</h1>\n<p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u5b9e\u73b0<a href=\"https://arxiv.org/abs/1607.08022\">\u5b9e\u4f8b\u89c4\u8303\u5316\uff1a\u5feb\u901f\u98ce\u683c\u5316\u7684\u7f3a\u5931\u6210\u5206</a>\u3002</p>\n<p>\u5f15\u5165\u4e86\u5b9e\u4f8b\u89c4\u8303\u5316\u4ee5\u6539\u8fdb<a href=\"https://paperswithcode.com/task/style-transfer\">\u6837\u5f0f\u4f20\u8f93</a>\u3002\u5b83\u57fa\u4e8e\u8fd9\u6837\u7684\u89c2\u5bdf\uff0c\u5373\u98ce\u683c\u5316\u4e0d\u5e94\u4f9d\u8d56\u4e8e\u5185\u5bb9\u56fe\u50cf\u7684\u5bf9\u6bd4\u5ea6\u3002\u201c\u5bf9\u6bd4\u5ea6\u6807\u51c6\u5316\u201d \u662f</p>\n<p><span translate=no>_^_0_^_</span></p>\n<p>\u5176\u4e2d\uff0c<span translate=no>_^_1_^_</span>\u662f\u4e00\u6279\u5177\u6709\u5c3a\u5bf8\u56fe\u50cf\u7d22\u5f15<span translate=no>_^_2_^_</span>\u3001\u7279\u5f81\u901a\u9053<span translate=no>_^_3_^_</span>\u548c\u7a7a\u95f4\u4f4d\u7f6e\u7684\u56fe\u50cf<span translate=no>_^_4_^_</span>\u3002</p>\n<p>\u7531\u4e8e\u5377\u79ef\u7f51\u7edc\u5f88\u96be\u5b66\u4e60 \u201c\u5bf9\u6bd4\u5ea6\u5f52\u4e00\u5316\u201d\uff0c\u672c\u6587\u4ecb\u7ecd\u4e86\u5b9e\u4f8b\u89c4\u8303\u5316\u6765\u505a\u5230\u8fd9\u4e00\u70b9\u3002</p>\n<p>\u4ee5\u4e0b\u662f\u4f7f\u7528\u5b9e\u4f8b\u89c4\u8303\u5316\u7684 <a href=\"experiment.html\">CIFAR 10 \u5206\u7c7b\u6a21\u578b</a>\u3002</p>\n",
|
||||
"<h2>Instance Normalization Layer</h2>\n<p>Instance normalization layer <span translate=no>_^_0_^_</span> normalizes the input <span translate=no>_^_1_^_</span> as follows:</p>\n<p>When input <span translate=no>_^_2_^_</span> is a batch of image representations, where <span translate=no>_^_3_^_</span> is the batch size, <span translate=no>_^_4_^_</span> is the number of channels, <span translate=no>_^_5_^_</span> is the height and <span translate=no>_^_6_^_</span> is the width. <span translate=no>_^_7_^_</span> and <span translate=no>_^_8_^_</span>. The affine transformation with <span translate=no>_^_9_^_</span> and <span translate=no>_^_10_^_</span> are optional.</p>\n<p><span translate=no>_^_11_^_</span></p>\n": "<h2>\u5b9e\u4f8b\u89c4\u8303\u5316\u5c42</h2>\n<p>\u5b9e\u4f8b\u5f52\u4e00\u5316\u5c42\u5c06\u8f93\u5165<span translate=no>_^_0_^_</span>\u5f52\u4e00\u5316\uff0c<span translate=no>_^_1_^_</span>\u5982\u4e0b\u6240\u793a\uff1a</p>\n<p>\u5f53\u8f93\u5165<span translate=no>_^_2_^_</span>\u662f\u4e00\u6279\u56fe\u50cf\u8868\u793a\u65f6\uff0c\u5176\u4e2d<span translate=no>_^_3_^_</span>\u662f\u6279\u6b21\u5927\u5c0f\uff0c<span translate=no>_^_4_^_</span>\u662f\u901a\u9053\u6570\uff0c<span translate=no>_^_5_^_</span>\u662f\u9ad8\u5ea6\u548c<span translate=no>_^_6_^_</span>\u662f\u5bbd\u5ea6\u3002<span translate=no>_^_7_^_</span>\u548c<span translate=no>_^_8_^_</span>\u3002\u5e26<span translate=no>_^_9_^_</span>\u548c\u7684\u4eff\u5c04\u53d8\u6362<span translate=no>_^_10_^_</span>\u662f\u53ef\u9009\u7684\u3002</p>\n<p><span translate=no>_^_11_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> denotes any number of (possibly 0) dimensions. For example, in an image (2D) convolution this will be <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u5f20\u91cf<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u8868\u793a\u4efb\u610f\u6570\u91cf\uff08\u53ef\u80fd\u4e3a 0\uff09\u7684\u7ef4\u5ea6\u3002\u4f8b\u5982\uff0c\u5728\u56fe\u50cf\uff082D\uff09\u5377\u79ef\u4e2d\uff0c\u8fd9\u5c06\u662f<span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p> Simple test</p>\n": "<p>\u7b80\u5355\u6d4b\u8bd5</p>\n",
|
||||
"<p>Calculate the mean across last dimension i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684\u5e73\u5747\u503c\uff0c\u5373\u6bcf\u4e2a\u8981\u7d20\u7684\u5747\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean across first and last dimension; i.e. the means for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u7b2c\u4e00\u7ef4\u548c\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684\u5747\u65b9\u503c\uff1b\u5373\u6bcf\u4e2a\u8981\u7d20\u7684\u5747\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create parameters for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for scale and shift </p>\n": "<p><span translate=no>_^_1_^_</span>\u4e3a\u7f29\u653e<span translate=no>_^_0_^_</span>\u548c\u79fb\u4f4d\u521b\u5efa\u53c2\u6570</p>\n",
|
||||
"<p>Get the batch size </p>\n": "<p>\u83b7\u53d6\u6279\u6b21\u5927\u5c0f</p>\n",
|
||||
"<p>Keep the original shape </p>\n": "<p>\u4fdd\u6301\u539f\u59cb\u5f62\u72b6</p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u89c4\u8303\u5316<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Reshape into <span translate=no>_^_0_^_</span> </p>\n": "<p>\u91cd\u5851\u6210<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Reshape to original and return </p>\n": "<p>\u91cd\u5851\u4e3a\u539f\u59cb\u5f62\u72b6\u7136\u540e\u8fd4\u56de</p>\n",
|
||||
"<p>Sanity check to make sure the number of features is the same </p>\n": "<p>\u8fdb\u884c\u5065\u5168\u6027\u68c0\u67e5\u4ee5\u786e\u4fdd\u8981\u7d20\u6570\u91cf\u76f8\u540c</p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7f29\u653e\u548c\u79fb\u52a8<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Variance for each feature <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6bcf\u4e2a\u8981\u7d20\u7684\u65b9\u5dee<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in the input </li>\n<li><span translate=no>_^_1_^_</span> is <span translate=no>_^_2_^_</span>, used in <span translate=no>_^_3_^_</span> for numerical stability </li>\n<li><span translate=no>_^_4_^_</span> is whether to scale and shift the normalized value</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8f93\u5165\u4e2d\u7684\u8981\u7d20\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f<span translate=no>_^_2_^_</span>\uff0c<span translate=no>_^_3_^_</span>\u7528\u4e8e\u6570\u503c\u7a33\u5b9a\u6027</li>\n<li><span translate=no>_^_4_^_</span>\u662f\u5426\u7f29\u653e\u548c\u79fb\u52a8\u5f52\u4e00\u5316\u503c</li></ul>\n",
|
||||
"A PyTorch implementation/tutorial of instance normalization.": "\u4e00\u4e2a\u5173\u4e8e\u5b9e\u4f8b\u89c4\u8303\u5316\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002",
|
||||
"Instance Normalization": "\u5b9e\u4f8b\u89c4\u8303\u5316"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Instance Normalization</h1>\n<p>This demonstrates the use of an instance normalization layer in a convolutional neural network for classification. Not that instance normalization was designed for style transfer and this is only a demo.</p>\n": "<h1>\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306e\u6b63\u898f\u5316\u306e\u305f\u3081\u306e CIFAR10 \u306e\u5b9f\u9a13</h1>\n<p>\u3053\u308c\u306f\u3001\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316\u5c64\u3092\u5206\u985e\u306b\u4f7f\u7528\u3057\u3066\u3044\u308b\u3053\u3068\u3092\u793a\u3057\u3066\u3044\u307e\u3059\u3002\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306e\u6b63\u898f\u5316\u304c\u30b9\u30bf\u30a4\u30eb\u8ee2\u9001\u306e\u305f\u3081\u306b\u8a2d\u8a08\u3055\u308c\u305f\u308f\u3051\u3067\u306f\u306a\u304f\u3001\u3053\u308c\u306f\u5358\u306a\u308b\u30c7\u30e2\u3067\u3059</p>\u3002\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u30e2\u30c7\u30eb\u4f5c\u6210</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>CIFAR-10 \u5206\u985e\u7528\u306e VGG \u30e2\u30c7\u30eb</h3>\n<p><a href=\"../../experiments/cifar10.html\">\u3053\u308c\u306f\u4e00\u822c\u7684\u306a VGG \u30b9\u30bf\u30a4\u30eb\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306b\u7531\u6765\u3057\u307e\u3059</a>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u69cb\u6210\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3092\u5b9f\u884c\u3057\u307e\u3059</p>\n",
|
||||
"CIFAR10 Experiment to try Instance Normalization": "\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306e\u6b63\u898f\u5316\u3092\u8a66\u3059\u305f\u3081\u306e CIFAR10 \u5b9f\u9a13",
|
||||
"This trains is a simple convolutional neural network that uses instance normalization to classify CIFAR10 images.": "\u3053\u306e\u30c8\u30ec\u30a4\u30f3\u306f\u3001\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306e\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3057\u3066 CIFAR10 \u753b\u50cf\u3092\u5206\u985e\u3059\u308b\u5358\u7d14\u306a\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Instance Normalization</h1>\n<p>This demonstrates the use of an instance normalization layer in a convolutional neural network for classification. Not that instance normalization was designed for style transfer and this is only a demo.</p>\n": "<h1>\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca\u0dbd\u0dd9\u0dc3 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf CIFAR10 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0d82\u0dc0\u0dc4\u0db1 \u0dc3\u0dca\u0db1\u0dcf\u0dba\u0dd4\u0d9a \u0da2\u0dcf\u0dbd\u0dba\u0d9a \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab \u0dc3\u0dca\u0dad\u0dbb\u0dba\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db8\u0dd9\u0dba\u0dd2\u0db1\u0dca \u0db4\u0dd9\u0db1\u0dca\u0db1\u0dd4\u0db8\u0dca \u0d9a\u0dd9\u0dbb\u0dda. \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0db1\u0ddc\u0dc0\u0dda \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db8\u0ddd\u0dc3\u0dca\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0db8\u0dd9\u0dba \u0db1\u0dd2\u0dbb\u0dd6\u0db4\u0dab\u0dba\u0d9a\u0dca \u0db4\u0db8\u0dab\u0dd2. </p>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>CIFA-10\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf VGG \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h3>\n<p>\u0db8\u0dd9\u0dba <a href=\"../../experiments/cifar10.html\">\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba VGG \u0dc0\u0dd2\u0dbd\u0dcf\u0dc3\u0dd2\u0dad\u0dcf\u0dc0\u0dda \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0dd9\u0db1\u0dca</a>\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dd3. </p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"CIFAR10 Experiment to try Instance Normalization": "\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0da7 CIFAR10 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This trains is a simple convolutional neural network that uses instance normalization to classify CIFAR10 images.": "\u0db8\u0dd9\u0db8 \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba CIFAR10 \u0dbb\u0dd6\u0db4 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc3\u0dbb\u0dbd \u0dc3\u0d82\u0da0\u0dbd\u0dd2\u0dad \u0dc3\u0dca\u0db1\u0dcf\u0dba\u0dd4\u0d9a \u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment for Instance Normalization</h1>\n<p>This demonstrates the use of an instance normalization layer in a convolutional neural network for classification. Not that instance normalization was designed for style transfer and this is only a demo.</p>\n": "<h1>CIFAR10 \u5b9e\u4f8b\u89c4\u8303\u5316\u5b9e\u9a8c</h1>\n<p>\u8fd9\u6f14\u793a\u4e86\u5982\u4f55\u5728\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\u4e2d\u4f7f\u7528\u5b9e\u4f8b\u5f52\u4e00\u5316\u5c42\u8fdb\u884c\u5206\u7c7b\u3002\u5e76\u4e0d\u662f\u8bf4\u5b9e\u4f8b\u89c4\u8303\u5316\u662f\u4e3a\u98ce\u683c\u8f6c\u79fb\u800c\u8bbe\u8ba1\u7684\uff0c\u8fd9\u53ea\u662f\u4e00\u4e2a\u6f14\u793a\u3002</p>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u521b\u5efa\u6a21\u578b</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>\u7528\u4e8e CIFAR-10 \u5206\u7c7b\u7684 VGG \u6a21\u578b</h3>\n<p>\u8fd9\u6e90\u4e8e<a href=\"../../experiments/cifar10.html\">\u901a\u7528\u7684 VGG \u98ce\u683c\u67b6\u6784</a>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c\u5e76\u8fd0\u884c\u8bad\u7ec3\u5faa\u73af</p>\n",
|
||||
"CIFAR10 Experiment to try Instance Normalization": "CIFAR10 \u5c1d\u8bd5\u5b9e\u4f8b\u89c4\u8303\u5316\u7684\u5b9e\u9a8c",
|
||||
"This trains is a simple convolutional neural network that uses instance normalization to classify CIFAR10 images.": "\u8be5\u5217\u8f66\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff0c\u5b83\u4f7f\u7528\u5b9e\u4f8b\u5f52\u4e00\u5316\u5bf9 CIFAR10 \u56fe\u50cf\u8fdb\u884c\u5206\u7c7b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/instance_norm/index.html\">Instance Normalization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.08022\">Instance Normalization: The Missing Ingredient for Fast Stylization</a>.</p>\n<p>Instance normalization was introduced to improve <a href=\"https://paperswithcode.com/task/style-transfer\">style transfer</a>. It is based on the observation that stylization should not depend on the contrast of the content image. Since it's hard for a convolutional network to learn "contrast normalization", this paper introduces instance normalization which does that.</p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/instance_norm/index.html\">\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316</a></h1>\n<p>\u3053\u308c\u306f <a href=\"https://pytorch.org\">PyTorch</a> \u306e\u300c<a href=\"https://arxiv.org/abs/1607.08022\">\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316:\u9ad8\u901f\u30b9\u30bf\u30a4\u30eb\u5316\u306b\u6b20\u3051\u3066\u3044\u308b\u8981\u7d20</a>\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002</p>\n<p><a href=\"https://paperswithcode.com/task/style-transfer\">\u30b9\u30bf\u30a4\u30eb\u8ee2\u9001\u3092\u6539\u5584\u3059\u308b\u305f\u3081\u306b\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306e\u6b63\u898f\u5316\u304c\u5c0e\u5165\u3055\u308c\u307e\u3057\u305f</a>\u3002\u3053\u308c\u306f\u3001\u30b9\u30bf\u30a4\u30eb\u8a2d\u5b9a\u306f\u30b3\u30f3\u30c6\u30f3\u30c4\u753b\u50cf\u306e\u30b3\u30f3\u30c8\u30e9\u30b9\u30c8\u306b\u4f9d\u5b58\u3059\u3079\u304d\u3067\u306f\u306a\u3044\u3068\u3044\u3046\u89b3\u5bdf\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059\u3002\u7573\u307f\u8fbc\u307f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u306f\u300c\u30b3\u30f3\u30c8\u30e9\u30b9\u30c8\u6b63\u898f\u5316\u300d\u3092\u5b66\u7fd2\u3059\u308b\u306e\u306f\u96e3\u3057\u3044\u306e\u3067\u3001\u672c\u7a3f\u3067\u306f\u305d\u308c\u3092\u884c\u3046\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316\u3092\u7d39\u4ecb\u3057\u307e\u3059</p>\u3002\n",
|
||||
"Instance Normalization": "\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u6b63\u898f\u5316"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/instance_norm/index.html\">Instance Normalization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.08022\">Instance Normalization: The Missing Ingredient for Fast Stylization</a>.</p>\n<p>Instance normalization was introduced to improve <a href=\"https://paperswithcode.com/task/style-transfer\">style transfer</a>. It is based on the observation that stylization should not depend on the contrast of the content image. Since it's hard for a convolutional network to learn "contrast normalization", this paper introduces instance normalization which does that.</p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/instance_norm/index.html\">\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba</a></h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <a href=\"https://arxiv.org/abs/1607.08022\">\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba: \u0dc0\u0dda\u0d9c\u0dc0\u0dad\u0dca \u0dc1\u0ddb\u0dbd\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0dad\u0dd4\u0dbb\u0dd4\u0daf\u0dc4\u0db1\u0dca \u0dc0\u0dd6 \u0d85\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dc0\u0dca\u0dba\u0dba</a> . </p>\n<p><a href=\"https://paperswithcode.com/task/style-transfer\">\u0dc1\u0ddb\u0dbd\u0dd2\u0dba \u0db8\u0dcf\u0dbb\u0dd4</a>\u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0db1 \u0dbd\u0daf\u0dd3. \u0d91\u0dba \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd3 \u0d87\u0dad\u0dca\u0dad\u0dda \u0d85\u0db1\u0dca\u0dad\u0dbb\u0dca\u0d9c\u0dad \u0dbb\u0dd6\u0db4\u0dba\u0dda \u0dc0\u0dd9\u0db1\u0dc3 \u0db8\u0dad \u0dc1\u0ddb\u0dbd\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0dbb\u0db3\u0dcf \u0db1\u0ddc\u0dad\u0dd2\u0db6\u0dd2\u0dba \u0dba\u0dd4\u0dad\u0dd4 \u0db6\u0dc0 \u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db8\u0dad \u0dba. \u0dc3\u0db8\u0dca\u0db8\u0dd4\u0dad\u0dd2 \u0da2\u0dcf\u0dbd\u0dba\u0d9a\u0da7 \u201c\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc0\u0dd2\u0dbb\u0dd4\u0daf\u0dca\u0db0 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u201d \u0d89\u0d9c\u0dd9\u0db1 \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0daf\u0dd4\u0dc2\u0dca\u0d9a\u0dbb \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca, \u0db8\u0dd9\u0db8 \u0dbd\u0dd2\u0db4\u0dd2\u0dba \u0db8\u0d9f\u0dd2\u0db1\u0dca \u0d91\u0dba \u0dc3\u0dd2\u0daf\u0dd4 \u0d9a\u0dbb\u0db1 \u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc4\u0db3\u0dd4\u0db1\u0dca\u0dc0\u0dcf \u0daf\u0dd9\u0dba\u0dd2. </p>\n",
|
||||
"Instance Normalization": "\u0d8b\u0daf\u0dcf\u0dc4\u0dbb\u0dab\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/instance_norm/index.html\">Instance Normalization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.08022\">Instance Normalization: The Missing Ingredient for Fast Stylization</a>.</p>\n<p>Instance normalization was introduced to improve <a href=\"https://paperswithcode.com/task/style-transfer\">style transfer</a>. It is based on the observation that stylization should not depend on the contrast of the content image. Since it's hard for a convolutional network to learn "contrast normalization", this paper introduces instance normalization which does that.</p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/instance_norm/index.html\">\u5b9e\u4f8b\u89c4\u8303\u5316</a></h1>\n<p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u5b9e\u73b0<a href=\"https://arxiv.org/abs/1607.08022\">\u5b9e\u4f8b\u89c4\u8303\u5316\uff1a\u5feb\u901f\u98ce\u683c\u5316\u7684\u7f3a\u5931\u6210\u5206</a>\u3002</p>\n<p>\u5f15\u5165\u4e86\u5b9e\u4f8b\u89c4\u8303\u5316\u4ee5\u6539\u8fdb<a href=\"https://paperswithcode.com/task/style-transfer\">\u6837\u5f0f\u4f20\u8f93</a>\u3002\u5b83\u57fa\u4e8e\u8fd9\u6837\u7684\u89c2\u5bdf\uff0c\u5373\u98ce\u683c\u5316\u4e0d\u5e94\u4f9d\u8d56\u4e8e\u5185\u5bb9\u56fe\u50cf\u7684\u5bf9\u6bd4\u5ea6\u3002\u7531\u4e8e\u5377\u79ef\u7f51\u7edc\u5f88\u96be\u5b66\u4e60 \u201c\u5bf9\u6bd4\u5ea6\u5f52\u4e00\u5316\u201d\uff0c\u672c\u6587\u4ecb\u7ecd\u4e86\u5b9e\u4f8b\u89c4\u8303\u5316\u6765\u505a\u5230\u8fd9\u4e00\u70b9\u3002</p>\n",
|
||||
"Instance Normalization": "\u5b9e\u4f8b\u89c4\u8303\u5316"
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"<h1>Layer Normalization</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.06450\">Layer Normalization</a>.</p>\n<h3>Limitations of <a href=\"../batch_norm/index.html\">Batch Normalization</a></h3>\n<ul><li>You need to maintain running means. </li>\n<li>Tricky for RNNs. Do you need different normalizations for each step? </li>\n<li>Doesn't work with small batch sizes; large NLP models are usually trained with small batch sizes. </li>\n<li>Need to compute means and variances across devices in distributed training.</li></ul>\n<h2>Layer Normalization</h2>\n<p>Layer normalization is a simpler normalization method that works on a wider range of settings. Layer normalization transforms the inputs to have zero mean and unit variance across the features. <em>Note that batch normalization fixes the zero mean and unit variance for each element.</em> Layer normalization does it for each batch across all elements.</p>\n<p>Layer normalization is generally used for NLP tasks.</p>\n<p>We have used layer normalization in most of the <a href=\"../../transformers/gpt/index.html\">transformer implementations</a>.</p>\n": "<h1>\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316</h1>\n<p><a href=\"https://arxiv.org/abs/1607.06450\">\u3053\u308c\u306f\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316\u306e</a> <a href=\"https://pytorch.org\">PyTorch</a> \u5b9f\u88c5\u3067\u3059\u3002</p>\n<h3><a href=\"../batch_norm/index.html\">\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306e\u5236\u9650\u4e8b\u9805</a></h3>\n<ul><li>\u30e9\u30f3\u30cb\u30f3\u30b0\u624b\u6bb5\u3092\u7dad\u6301\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002</li>\n<li>RNN\u306b\u3068\u3063\u3066\u306f\u6271\u3044\u306b\u304f\u3044\u3002\u30b9\u30c6\u30c3\u30d7\u3054\u3068\u306b\u7570\u306a\u308b\u6b63\u898f\u5316\u304c\u5fc5\u8981\u3067\u3059\u304b</li>?\n<li>\u5c0f\u3055\u306a\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3067\u306f\u6a5f\u80fd\u3057\u307e\u305b\u3093\u3002\u5927\u898f\u6a21\u306aNLP\u30e2\u30c7\u30eb\u306f\u901a\u5e38\u3001\u5c0f\u3055\u306a\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3067\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3055\u308c\u307e\u3059\u3002</li>\n</ul><li>\u5206\u6563\u578b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3067\u306f\u3001\u30c7\u30d0\u30a4\u30b9\u9593\u306e\u5e73\u5747\u3068\u5206\u6563\u3092\u8a08\u7b97\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002</li>\n<h2>\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316</h2>\n<p>\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316\u306f\u3001\u3088\u308a\u5e45\u5e83\u3044\u8a2d\u5b9a\u306b\u9069\u7528\u3067\u304d\u308b\u3001\u3088\u308a\u5358\u7d14\u306a\u6b63\u898f\u5316\u65b9\u6cd5\u3067\u3059\u3002\u5c64\u306e\u6b63\u898f\u5316\u306b\u3088\u308a\u3001\u5165\u529b\u306f\u7279\u5fb4\u5168\u4f53\u3067\u5e73\u5747\u304c\u30bc\u30ed\u3067\u5358\u4f4d\u5206\u6563\u304c\u306a\u304f\u306a\u308b\u3088\u3046\u306b\u5909\u63db\u3055\u308c\u307e\u3059\u3002<em>\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3067\u306f\u3001\u5404\u8981\u7d20\u306e\u30bc\u30ed\u5e73\u5747\u3068\u5358\u4f4d\u5206\u6563\u304c\u56fa\u5b9a\u3055\u308c\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002</em>\u30ec\u30a4\u30e4\u30fc\u306e\u6b63\u898f\u5316\u306f\u3001\u3059\u3079\u3066\u306e\u8981\u7d20\u306e\u30d0\u30c3\u30c1\u3054\u3068\u306b\u6b63\u898f\u5316\u3092\u884c\u3044\u307e\u3059</p>\u3002\n<p>\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316\u306f\u901a\u5e38\u3001NLP \u30bf\u30b9\u30af\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059\u3002</p>\n<p><a href=\"../../transformers/gpt/index.html\">\u307b\u3068\u3093\u3069\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u5b9f\u88c5\u3067\u5c64\u306e\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h2>Layer Normalization</h2>\n<p>Layer normalization <span translate=no>_^_0_^_</span> normalizes the input <span translate=no>_^_1_^_</span> as follows:</p>\n<p>When input <span translate=no>_^_2_^_</span> is a batch of embeddings, where <span translate=no>_^_3_^_</span> is the batch size and <span translate=no>_^_4_^_</span> is the number of features. <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>. <span translate=no>_^_7_^_</span></p>\n<p>When input <span translate=no>_^_8_^_</span> is a batch of a sequence of embeddings, where <span translate=no>_^_9_^_</span> is the batch size, <span translate=no>_^_10_^_</span> is the number of channels, <span translate=no>_^_11_^_</span> is the length of the sequence. <span translate=no>_^_12_^_</span> and <span translate=no>_^_13_^_</span>. <span translate=no>_^_14_^_</span></p>\n<p>When input <span translate=no>_^_15_^_</span> is a batch of image representations, where <span translate=no>_^_16_^_</span> is the batch size, <span translate=no>_^_17_^_</span> is the number of channels, <span translate=no>_^_18_^_</span> is the height and <span translate=no>_^_19_^_</span> is the width. This is not a widely used scenario. <span translate=no>_^_20_^_</span> and <span translate=no>_^_21_^_</span>. <span translate=no>_^_22_^_</span></p>\n": "<h2>\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316</h2>\n<p><span translate=no>_^_0_^_</span>\u5c64\u306e\u6b63\u898f\u5316\u306f\u3001<span translate=no>_^_1_^_</span>\u5165\u529b\u3092\u6b21\u306e\u3088\u3046\u306b\u6b63\u898f\u5316\u3057\u307e\u3059\u3002</p>\n<p>\u5165\u529b\u304c\u57cb\u3081\u8fbc\u307f\u306e\u30d0\u30c3\u30c1\u306e\u5834\u5408\u3001<span translate=no>_^_2_^_</span>\u306f\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3001<span translate=no>_^_3_^_</span><span translate=no>_^_4_^_</span>\u306f\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u6570\u3067\u3059\u3002<span translate=no>_^_5_^_</span>\u3068<span translate=no>_^_6_^_</span>\u3002<span translate=no>_^_7_^_</span></p>\n<p><span translate=no>_^_8_^_</span>\u5165\u529b\u304c\u57cb\u3081\u8fbc\u307f\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u30d0\u30c3\u30c1\u3067\u3042\u308b\u5834\u5408\u3001<span translate=no>_^_9_^_</span>\u306f\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3001<span translate=no>_^_10_^_</span>\u306f\u30c1\u30e3\u30cd\u30eb\u6570\u3001<span translate=no>_^_11_^_</span>\u306f\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3067\u3059\u3002<span translate=no>_^_12_^_</span>\u3068<span translate=no>_^_13_^_</span>\u3002<span translate=no>_^_14_^_</span></p>\n<p><span translate=no>_^_15_^_</span>\u5165\u529b\u304c\u30a4\u30e1\u30fc\u30b8\u8868\u73fe\u306e\u30d0\u30c3\u30c1\u306e\u5834\u5408\u3001<span translate=no>_^_16_^_</span>\u306f\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3001<span translate=no>_^_17_^_</span>\u306f\u30c1\u30e3\u30cd\u30eb\u6570\u3001<span translate=no>_^_18_^_</span>\u306f\u9ad8\u3055\u3001<span translate=no>_^_19_^_</span>\u306f\u5e45\u3067\u3059\u3002\u3053\u308c\u306f\u3042\u307e\u308a\u4f7f\u308f\u308c\u3066\u3044\u306a\u3044\u30b7\u30ca\u30ea\u30aa\u3067\u3059\u3002<span translate=no>_^_20_^_</span>\u3068<span translate=no>_^_21_^_</span>\u3002<span translate=no>_^_22_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> could be any number of dimensions. For example, in an NLP task this will be <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u5f62\u72b6\u306e\u30c6\u30f3\u30bd\u30eb\u3067\u3059\u3002<span translate=no>_^_2_^_</span>\u6b21\u5143\u306f\u3044\u304f\u3064\u3067\u3082\u304b\u307e\u3044\u307e\u305b\u3093\u3002\u305f\u3068\u3048\u3070\u3001NLP \u30bf\u30b9\u30af\u3067\u306f\u3001\u6b21\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059</p>\u3002<span translate=no>_^_3_^_</span>\n",
|
||||
"<p> Simple test</p>\n": "<p>\u7c21\u5358\u306a\u30c6\u30b9\u30c8</p>\n",
|
||||
"<p>Calculate the mean of all elements; i.e. the means for each element <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3059\u3079\u3066\u306e\u8981\u7d20\u306e\u5e73\u5747\u3001\u3064\u307e\u308a\u5404\u8981\u7d20\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean of all elements; i.e. the means for each element <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3059\u3079\u3066\u306e\u8981\u7d20\u306e\u4e8c\u4e57\u5e73\u5747\u3001\u3064\u307e\u308a\u5404\u8981\u7d20\u306e\u5e73\u5747\u3092\u8a08\u7b97\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Convert <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u306b\u5909\u63db <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Create parameters for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for gain and bias </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u30b2\u30a4\u30f3\u3068\u30d0\u30a4\u30a2\u30b9\u306e\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3068\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30ce\u30fc\u30de\u30e9\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Sanity check to make sure the shapes match </p>\n": "<p>\u5f62\u72b6\u304c\u5408\u3063\u3066\u3044\u308b\u304b\u78ba\u8a8d\u3059\u308b\u30b5\u30cb\u30c6\u30a3\u30c1\u30a7\u30c3\u30af</p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30b1\u30fc\u30eb\u3068\u30b7\u30d5\u30c8 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The dimensions to calculate the mean and variance on </p>\n": "<p>\u5e73\u5747\u3068\u5206\u6563\u3092\u8a08\u7b97\u3059\u308b\u5bfe\u8c61\u306e\u30c7\u30a3\u30e1\u30f3\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Variance of all element <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5168\u8981\u7d20\u306e\u5dee\u7570 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> is the shape of the elements (except the batch). The input should then be <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is <span translate=no>_^_4_^_</span>, used in <span translate=no>_^_5_^_</span> for numerical stability </li>\n<li><span translate=no>_^_6_^_</span> is whether to scale and shift the normalized value</li></ul>\n<p>We've tried to use the same names for arguments as PyTorch <span translate=no>_^_7_^_</span> implementation.</p>\n": "<ul><li><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u8981\u7d20\u306e\u5f62\u72b6\u3067\u3059 (\u30d0\u30c3\u30c1\u306f\u9664\u304f)\u3002\u305d\u306e\u5834\u5408\u3001\u5165\u529b\u306f\u6b21\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span><span translate=no>_^_5_^_</span>\u6570\u5024\u306e\u5b89\u5b9a\u6027\u306e\u305f\u3081\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059 <span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_6_^_</span>\u6b63\u898f\u5316\u3055\u308c\u305f\u5024\u3092\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3057\u3066\u30b7\u30d5\u30c8\u3059\u308b\u304b\u3069\u3046\u304b\u3067\u3059</li></ul>\n<p>\u5f15\u6570\u306b\u306f PyTorch <span translate=no>_^_7_^_</span> \u5b9f\u88c5\u3068\u540c\u3058\u540d\u524d\u3092\u4f7f\u7528\u3057\u3088\u3046\u3068\u3057\u307e\u3057\u305f\u3002</p>\n",
|
||||
"A PyTorch implementation/tutorial of layer normalization.": "\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316\u306ePyTorch\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"Layer Normalization": "\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"<h1>Layer Normalization</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.06450\">Layer Normalization</a>.</p>\n<h3>Limitations of <a href=\"../batch_norm/index.html\">Batch Normalization</a></h3>\n<ul><li>You need to maintain running means. </li>\n<li>Tricky for RNNs. Do you need different normalizations for each step? </li>\n<li>Doesn't work with small batch sizes; large NLP models are usually trained with small batch sizes. </li>\n<li>Need to compute means and variances across devices in distributed training.</li></ul>\n<h2>Layer Normalization</h2>\n<p>Layer normalization is a simpler normalization method that works on a wider range of settings. Layer normalization transforms the inputs to have zero mean and unit variance across the features. <em>Note that batch normalization fixes the zero mean and unit variance for each element.</em> Layer normalization does it for each batch across all elements.</p>\n<p>Layer normalization is generally used for NLP tasks.</p>\n<p>We have used layer normalization in most of the <a href=\"../../transformers/gpt/index.html\">transformer implementations</a>.</p>\n": "<h1>\u5c42\u89c4\u8303\u5316</h1>\n<p>\u8fd9\u662f<a href=\"https://arxiv.org/abs/1607.06450\">\u5c42\u89c4\u8303\u5316</a>\u7684 <a href=\"https://pytorch.org\">PyTorch</a> \u5b9e\u73b0\u3002</p>\n<h3><a href=\"../batch_norm/index.html\">\u6279\u91cf\u6807\u51c6\u5316\u7684</a>\u5c40\u9650\u6027</h3>\n<ul><li>\u4f60\u9700\u8981\u4fdd\u6301\u8dd1\u6b65\u624b\u6bb5\u3002</li>\n<li>\u5bf9\u4e8e RNN \u6765\u8bf4\u5f88\u68d8\u624b\u3002\u6bcf\u4e2a\u6b65\u9aa4\u90fd\u9700\u8981\u4e0d\u540c\u7684\u89c4\u8303\u5316\u5417\uff1f</li>\n<li>\u4e0d\u9002\u7528\u4e8e\u5c0f\u6279\u91cf\uff1b\u5927\u578b NLP \u6a21\u578b\u901a\u5e38\u4f7f\u7528\u5c0f\u6279\u91cf\u8fdb\u884c\u8bad\u7ec3\u3002</li>\n<li>\u9700\u8981\u5728\u5206\u5e03\u5f0f\u8bad\u7ec3\u4e2d\u8ba1\u7b97\u8bbe\u5907\u95f4\u7684\u5747\u503c\u548c\u65b9\u5dee\u3002</li></ul>\n<h2>\u5c42\u89c4\u8303\u5316</h2>\n<p>\u56fe\u5c42\u5f52\u4e00\u5316\u662f\u4e00\u79cd\u66f4\u7b80\u5355\u7684\u5f52\u4e00\u5316\u65b9\u6cd5\uff0c\u9002\u7528\u4e8e\u66f4\u5e7f\u6cdb\u7684\u8bbe\u7f6e\u3002\u56fe\u5c42\u5f52\u4e00\u5316\u4f1a\u5c06\u8f93\u5165\u53d8\u6362\u4e3a\u5404\u8981\u7d20\u7684\u5747\u503c\u548c\u5355\u4f4d\u65b9\u5dee\u4e3a\u96f6\u3002<em>\u8bf7\u6ce8\u610f\uff0c\u6279\u91cf\u5f52\u4e00\u5316\u4fee\u590d\u4e86\u6bcf\u4e2a\u5143\u7d20\u7684\u96f6\u5747\u503c\u548c\u5355\u4f4d\u65b9\u5dee\u3002</em>\u5c42\u5f52\u4e00\u5316\u5bf9\u6240\u6709\u5143\u7d20\u7684\u6bcf\u4e2a\u6279\u6b21\u6267\u884c\u6b64\u64cd\u4f5c\u3002</p>\n<p>\u5c42\u5f52\u4e00\u5316\u901a\u5e38\u7528\u4e8e NLP \u4efb\u52a1\u3002</p>\n<p>\u6211\u4eec\u5728\u5927\u591a\u6570<a href=\"../../transformers/gpt/index.html\">\u53d8\u538b\u5668\u5b9e\u73b0</a>\u4e2d\u90fd\u4f7f\u7528\u4e86\u5c42\u5f52\u4e00\u5316\u3002</p>\n",
|
||||
"<h2>Layer Normalization</h2>\n<p>Layer normalization <span translate=no>_^_0_^_</span> normalizes the input <span translate=no>_^_1_^_</span> as follows:</p>\n<p>When input <span translate=no>_^_2_^_</span> is a batch of embeddings, where <span translate=no>_^_3_^_</span> is the batch size and <span translate=no>_^_4_^_</span> is the number of features. <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span>. <span translate=no>_^_7_^_</span></p>\n<p>When input <span translate=no>_^_8_^_</span> is a batch of a sequence of embeddings, where <span translate=no>_^_9_^_</span> is the batch size, <span translate=no>_^_10_^_</span> is the number of channels, <span translate=no>_^_11_^_</span> is the length of the sequence. <span translate=no>_^_12_^_</span> and <span translate=no>_^_13_^_</span>. <span translate=no>_^_14_^_</span></p>\n<p>When input <span translate=no>_^_15_^_</span> is a batch of image representations, where <span translate=no>_^_16_^_</span> is the batch size, <span translate=no>_^_17_^_</span> is the number of channels, <span translate=no>_^_18_^_</span> is the height and <span translate=no>_^_19_^_</span> is the width. This is not a widely used scenario. <span translate=no>_^_20_^_</span> and <span translate=no>_^_21_^_</span>. <span translate=no>_^_22_^_</span></p>\n": "<h2>\u5c42\u89c4\u8303\u5316</h2>\n<p>\u56fe\u5c42<span translate=no>_^_0_^_</span>\u5f52\u4e00\u5316\u5c06\u8f93\u5165\u5f52\u4e00\u5316<span translate=no>_^_1_^_</span>\uff0c\u5982\u4e0b\u6240\u793a\uff1a</p>\n<p>\u5f53\u8f93\u5165<span translate=no>_^_2_^_</span>\u662f\u4e00\u6279\u5d4c\u5165\u65f6\uff0c\u5176\u4e2d<span translate=no>_^_3_^_</span>\u662f\u6279\u6b21\u5927\u5c0f\uff0c<span translate=no>_^_4_^_</span>\u662f\u8981\u7d20\u7684\u6570\u91cf\u3002<span translate=no>_^_5_^_</span>\u548c<span translate=no>_^_6_^_</span>\u3002<span translate=no>_^_7_^_</span></p>\n<p>\u5f53 input<span translate=no>_^_8_^_</span> \u662f\u5d4c\u5165\u5e8f\u5217\u4e2d\u7684\u4e00\u6279\u65f6\uff0c\u5176\u4e2d<span translate=no>_^_9_^_</span><span translate=no>_^_10_^_</span>\u662f\u6279\u6b21\u5927\u5c0f\uff0c<span translate=no>_^_11_^_</span>\u662f\u901a\u9053\u6570\uff0c\u662f\u987a\u5e8f\u3002<span translate=no>_^_12_^_</span>\u548c<span translate=no>_^_13_^_</span>\u3002<span translate=no>_^_14_^_</span></p>\n<p>\u5f53\u8f93\u5165<span translate=no>_^_15_^_</span>\u662f\u4e00\u6279\u56fe\u50cf\u8868\u793a\u65f6\uff0c\u5176\u4e2d<span translate=no>_^_16_^_</span>\u662f\u6279\u6b21\u5927\u5c0f\uff0c<span translate=no>_^_17_^_</span>\u662f\u901a\u9053\u6570\uff0c<span translate=no>_^_18_^_</span>\u662f\u9ad8\u5ea6\u548c<span translate=no>_^_19_^_</span>\u662f\u5bbd\u5ea6\u3002\u8fd9\u4e0d\u662f\u4e00\u4e2a\u5e7f\u6cdb\u4f7f\u7528\u7684\u573a\u666f\u3002<span translate=no>_^_20_^_</span>\u548c<span translate=no>_^_21_^_</span>\u3002<span translate=no>_^_22_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> is a tensor of shape <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> could be any number of dimensions. For example, in an NLP task this will be <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u5f20\u91cf<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u53ef\u4ee5\u662f\u4efb\u610f\u6570\u91cf\u7684\u7ef4\u5ea6\u3002\u4f8b\u5982\uff0c\u5728 NLP \u4efb\u52a1\u4e2d\uff0c\u8fd9\u5c06\u662f<span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p> Simple test</p>\n": "<p>\u7b80\u5355\u6d4b\u8bd5</p>\n",
|
||||
"<p>Calculate the mean of all elements; i.e. the means for each element <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u6240\u6709\u5143\u7d20\u7684\u5747\u503c\uff1b\u5373\u6bcf\u4e2a\u5143\u7d20\u7684\u5747\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate the squared mean of all elements; i.e. the means for each element <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u6240\u6709\u5143\u7d20\u7684\u5747\u65b9\u503c\uff1b\u5373\u6bcf\u4e2a\u5143\u7d20\u7684\u5747\u503c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Convert <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u8f6c\u6362<span translate=no>_^_0_^_</span>\u4e3a<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Create parameters for <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> for gain and bias </p>\n": "<p>\u4e3a\u589e\u76ca<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span>\u504f\u7f6e\u521b\u5efa\u53c2\u6570\u548c\u53c2\u6570</p>\n",
|
||||
"<p>Normalize <span translate=no>_^_0_^_</span> </p>\n": "<p>\u89c4\u8303\u5316<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Sanity check to make sure the shapes match </p>\n": "<p>\u8fdb\u884c\u5065\u5168\u6027\u68c0\u67e5\u4ee5\u786e\u4fdd\u5f62\u72b6\u5339\u914d</p>\n",
|
||||
"<p>Scale and shift <span translate=no>_^_0_^_</span> </p>\n": "<p>\u7f29\u653e\u548c\u79fb\u52a8<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>The dimensions to calculate the mean and variance on </p>\n": "<p>\u7528\u4e8e\u8ba1\u7b97\u5747\u503c\u548c\u65b9\u5dee\u7684\u7ef4\u5ea6</p>\n",
|
||||
"<p>Variance of all element <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6240\u6709\u5143\u7d20\u7684\u65b9\u5dee<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> is the shape of the elements (except the batch). The input should then be <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is <span translate=no>_^_4_^_</span>, used in <span translate=no>_^_5_^_</span> for numerical stability </li>\n<li><span translate=no>_^_6_^_</span> is whether to scale and shift the normalized value</li></ul>\n<p>We've tried to use the same names for arguments as PyTorch <span translate=no>_^_7_^_</span> implementation.</p>\n": "<ul><li><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u662f\u5143\u7d20\u7684\u5f62\u72b6\uff08\u6279\u6b21\u9664\u5916\uff09\u3002\u90a3\u4e48\u8f93\u5165\u5e94\u8be5\u662f<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f<span translate=no>_^_4_^_</span>\uff0c<span translate=no>_^_5_^_</span>\u7528\u4e8e\u6570\u503c\u7a33\u5b9a\u6027</li>\n<li><span translate=no>_^_6_^_</span>\u662f\u5426\u7f29\u653e\u548c\u79fb\u52a8\u5f52\u4e00\u5316\u503c</li></ul>\n<p>\u6211\u4eec\u5df2\u7ecf\u5c1d\u8bd5\u4f7f\u7528\u4e0e PyTorch<span translate=no>_^_7_^_</span> \u5b9e\u73b0\u76f8\u540c\u7684\u53c2\u6570\u540d\u79f0\u3002</p>\n",
|
||||
"A PyTorch implementation/tutorial of layer normalization.": "\u5173\u4e8e\u5c42\u89c4\u8303\u5316\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002",
|
||||
"Layer Normalization": "\u5c42\u89c4\u8303\u5316"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/layer_norm/index.html\">Layer Normalization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.06450\">Layer Normalization</a>.</p>\n<h3>Limitations of <a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">Batch Normalization</a></h3>\n<ul><li>You need to maintain running means. </li>\n<li>Tricky for RNNs. Do you need different normalizations for each step? </li>\n<li>Doesn't work with small batch sizes; large NLP models are usually trained with small batch sizes. </li>\n<li>Need to compute means and variances across devices in distributed training.</li></ul>\n<h2>Layer Normalization</h2>\n<p>Layer normalization is a simpler normalization method that works on a wider range of settings. Layer normalization transforms the inputs to have zero mean and unit variance across the features. <em>Note that batch normalization fixes the zero mean and unit variance for each element.</em> Layer normalization does it for each batch across all elements.</p>\n<p>Layer normalization is generally used for NLP tasks.</p>\n<p>We have used layer normalization in most of the <a href=\"https://nn.labml.ai/transformers/gpt/index.html\">transformer implementations</a>.</p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/layer_norm/index.html\">\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316</a></h1>\n<p><a href=\"https://arxiv.org/abs/1607.06450\">\u3053\u308c\u306f\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316\u306e</a> <a href=\"https://pytorch.org\">PyTorch</a> \u5b9f\u88c5\u3067\u3059\u3002</p>\n<h3><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u306e\u5236\u9650\u4e8b\u9805</a></h3>\n<ul><li>\u30e9\u30f3\u30cb\u30f3\u30b0\u624b\u6bb5\u3092\u7dad\u6301\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002</li>\n<li>RNN\u306b\u3068\u3063\u3066\u306f\u6271\u3044\u306b\u304f\u3044\u3002\u30b9\u30c6\u30c3\u30d7\u3054\u3068\u306b\u7570\u306a\u308b\u6b63\u898f\u5316\u304c\u5fc5\u8981\u3067\u3059\u304b</li>?\n<li>\u5c0f\u3055\u306a\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3067\u306f\u6a5f\u80fd\u3057\u307e\u305b\u3093\u3002\u5927\u898f\u6a21\u306aNLP\u30e2\u30c7\u30eb\u306f\u901a\u5e38\u3001\u5c0f\u3055\u306a\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3067\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3055\u308c\u307e\u3059\u3002</li>\n</ul><li>\u5206\u6563\u578b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3067\u306f\u3001\u30c7\u30d0\u30a4\u30b9\u9593\u306e\u5e73\u5747\u3068\u5206\u6563\u3092\u8a08\u7b97\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002</li>\n<h2>\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316</h2>\n<p>\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316\u306f\u3001\u3088\u308a\u5e45\u5e83\u3044\u8a2d\u5b9a\u306b\u9069\u7528\u3067\u304d\u308b\u3001\u3088\u308a\u5358\u7d14\u306a\u6b63\u898f\u5316\u65b9\u6cd5\u3067\u3059\u3002\u5c64\u306e\u6b63\u898f\u5316\u306b\u3088\u308a\u3001\u5165\u529b\u306f\u7279\u5fb4\u5168\u4f53\u3067\u5e73\u5747\u304c\u30bc\u30ed\u3067\u5358\u4f4d\u5206\u6563\u304c\u306a\u304f\u306a\u308b\u3088\u3046\u306b\u5909\u63db\u3055\u308c\u307e\u3059\u3002<em>\u30d0\u30c3\u30c1\u6b63\u898f\u5316\u3067\u306f\u3001\u5404\u8981\u7d20\u306e\u30bc\u30ed\u5e73\u5747\u3068\u5358\u4f4d\u5206\u6563\u304c\u56fa\u5b9a\u3055\u308c\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002</em>\u30ec\u30a4\u30e4\u30fc\u306e\u6b63\u898f\u5316\u306f\u3001\u3059\u3079\u3066\u306e\u8981\u7d20\u306e\u30d0\u30c3\u30c1\u3054\u3068\u306b\u6b63\u898f\u5316\u3092\u884c\u3044\u307e\u3059</p>\u3002\n<p>\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316\u306f\u901a\u5e38\u3001NLP \u30bf\u30b9\u30af\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059\u3002</p>\n<p><a href=\"https://nn.labml.ai/transformers/gpt/index.html\">\u307b\u3068\u3093\u3069\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u5b9f\u88c5\u3067\u5c64\u306e\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
|
||||
"Layer Normalization": "\u30ec\u30a4\u30e4\u30fc\u6b63\u898f\u5316"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/layer_norm/index.html\">Layer Normalization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1607.06450\">Layer Normalization</a>.</p>\n<h3>Limitations of <a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">Batch Normalization</a></h3>\n<ul><li>You need to maintain running means. </li>\n<li>Tricky for RNNs. Do you need different normalizations for each step? </li>\n<li>Doesn't work with small batch sizes; large NLP models are usually trained with small batch sizes. </li>\n<li>Need to compute means and variances across devices in distributed training.</li></ul>\n<h2>Layer Normalization</h2>\n<p>Layer normalization is a simpler normalization method that works on a wider range of settings. Layer normalization transforms the inputs to have zero mean and unit variance across the features. <em>Note that batch normalization fixes the zero mean and unit variance for each element.</em> Layer normalization does it for each batch across all elements.</p>\n<p>Layer normalization is generally used for NLP tasks.</p>\n<p>We have used layer normalization in most of the <a href=\"https://nn.labml.ai/transformers/gpt/index.html\">transformer implementations</a>.</p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/layer_norm/index.html\">\u5c42\u89c4\u8303\u5316</a></h1>\n<p>\u8fd9\u662f<a href=\"https://arxiv.org/abs/1607.06450\">\u5c42\u89c4\u8303\u5316</a>\u7684 <a href=\"https://pytorch.org\">PyTorch</a> \u5b9e\u73b0\u3002</p>\n<h3><a href=\"https://nn.labml.ai/normalization/batch_norm/index.html\">\u6279\u91cf\u6807\u51c6\u5316\u7684</a>\u5c40\u9650\u6027</h3>\n<ul><li>\u4f60\u9700\u8981\u4fdd\u6301\u8dd1\u6b65\u624b\u6bb5\u3002</li>\n<li>\u5bf9\u4e8e RNN \u6765\u8bf4\u5f88\u68d8\u624b\u3002\u6bcf\u4e2a\u6b65\u9aa4\u90fd\u9700\u8981\u4e0d\u540c\u7684\u89c4\u8303\u5316\u5417\uff1f</li>\n<li>\u4e0d\u9002\u7528\u4e8e\u5c0f\u6279\u91cf\uff1b\u5927\u578b NLP \u6a21\u578b\u901a\u5e38\u4f7f\u7528\u5c0f\u6279\u91cf\u8fdb\u884c\u8bad\u7ec3\u3002</li>\n<li>\u9700\u8981\u5728\u5206\u5e03\u5f0f\u8bad\u7ec3\u4e2d\u8ba1\u7b97\u8bbe\u5907\u95f4\u7684\u5747\u503c\u548c\u65b9\u5dee\u3002</li></ul>\n<h2>\u5c42\u89c4\u8303\u5316</h2>\n<p>\u56fe\u5c42\u5f52\u4e00\u5316\u662f\u4e00\u79cd\u66f4\u7b80\u5355\u7684\u5f52\u4e00\u5316\u65b9\u6cd5\uff0c\u9002\u7528\u4e8e\u66f4\u5e7f\u6cdb\u7684\u8bbe\u7f6e\u3002\u56fe\u5c42\u5f52\u4e00\u5316\u4f1a\u5c06\u8f93\u5165\u53d8\u6362\u4e3a\u5404\u8981\u7d20\u7684\u5747\u503c\u548c\u5355\u4f4d\u65b9\u5dee\u4e3a\u96f6\u3002<em>\u8bf7\u6ce8\u610f\uff0c\u6279\u91cf\u5f52\u4e00\u5316\u4fee\u590d\u4e86\u6bcf\u4e2a\u5143\u7d20\u7684\u96f6\u5747\u503c\u548c\u5355\u4f4d\u65b9\u5dee\u3002</em>\u5c42\u5f52\u4e00\u5316\u5bf9\u6240\u6709\u5143\u7d20\u7684\u6bcf\u4e2a\u6279\u6b21\u6267\u884c\u6b64\u64cd\u4f5c\u3002</p>\n<p>\u5c42\u5f52\u4e00\u5316\u901a\u5e38\u7528\u4e8e NLP \u4efb\u52a1\u3002</p>\n<p>\u6211\u4eec\u5728\u5927\u591a\u6570<a href=\"https://nn.labml.ai/transformers/gpt/index.html\">\u53d8\u538b\u5668\u5b9e\u73b0</a>\u4e2d\u90fd\u4f7f\u7528\u4e86\u5c42\u5f52\u4e00\u5316\u3002</p>\n",
|
||||
"Layer Normalization": "\u5c42\u89c4\u8303\u5316"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"2D Convolution Layer with Weight Standardization": "\u91cd\u307f\u6a19\u6e96\u5316\u3055\u308c\u305f 2D \u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc",
|
||||
"<h1>2D Convolution Layer with Weight Standardization</h1>\n<p>This is an implementation of a 2 dimensional convolution layer with <a href=\"./index.html\">Weight Standardization</a></p>\n": "<h1>\u91cd\u307f\u6a19\u6e96\u5316\u3055\u308c\u305f 2D \u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc</h1>\n<p><a href=\"./index.html\">\u3053\u308c\u306f\u3001\u91cd\u307f\u6a19\u6e96\u5316\u306b\u3088\u308b2\u6b21\u5143\u306e\u7573\u307f\u8fbc\u307f\u5c64\u306e\u5b9f\u88c5\u3067\u3059</a></p>\n",
|
||||
"<h2>2D Convolution Layer</h2>\n<p>This extends the standard 2D Convolution layer and standardize the weights before the convolution step.</p>\n": "<h2>2D \u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc</h2>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001\u6a19\u6e96\u306e2D\u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u304c\u62e1\u5f35\u3055\u308c\u3001\u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u30b9\u30c6\u30c3\u30d7\u306e\u524d\u306b\u30a6\u30a7\u30a4\u30c8\u304c\u6a19\u6e96\u5316\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p> A simple test to verify the tensor sizes</p>\n": "<p>\u30c6\u30f3\u30bd\u30eb\u30b5\u30a4\u30ba\u3092\u691c\u8a3c\u3059\u308b\u7c21\u5358\u306a\u30c6\u30b9\u30c8</p>\n",
|
||||
"A PyTorch implementation/tutorial of a 2D Convolution Layer with Weight Standardization.": "\u91cd\u307f\u6a19\u6e96\u5316\u306b\u3088\u308b2D\u30b3\u30f3\u30dc\u30ea\u30e5\u30fc\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u306ePyTorch\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002"
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"2D Convolution Layer with Weight Standardization": "\u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f 2D \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba",
|
||||
"<h1>2D Convolution Layer with Weight Standardization</h1>\n<p>This is an implementation of a 2 dimensional convolution layer with <a href=\"./index.html\">Weight Standardization</a></p>\n": "<h1>\u0db6\u0dbb\u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f 2D \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"./index.html\">\u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba</a>\u0dc3\u0db8\u0d9f 2 \u0db8\u0dcf\u0db1 \u0d9a\u0dd0\u0da7\u0dd2 \u0d9c\u0dd0\u0dc3\u0dd4\u0dab\u0dd4 \u0dad\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0d9a\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2</p>\n",
|
||||
"<h2>2D Convolution Layer</h2>\n<p>This extends the standard 2D Convolution layer and standardize the weights before the convolution step.</p>\n": "<h2>2D\u0dc3\u0db8\u0dca\u0db8\u0dd4\u0dad\u0dd2 \u0dc3\u0dca\u0dae\u0dbb\u0dba</h2>\n<p>\u0db8\u0dd9\u0dba\u0dc3\u0db8\u0dca\u0db8\u0dad 2D Convolution \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0db4\u0dd4\u0dc5\u0dd4\u0dbd\u0dca \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dbb \u0d9a\u0dd0\u0da7\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0da7 \u0db4\u0dd9\u0dbb \u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p> A simple test to verify the tensor sizes</p>\n": "<p> \u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0dad\u0dca\u0dba\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dbb\u0dbd \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba\u0d9a\u0dca</p>\n",
|
||||
"A PyTorch implementation/tutorial of a 2D Convolution Layer with Weight Standardization.": "\u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f 2D \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba."
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"2D Convolution Layer with Weight Standardization": "\u5177\u6709\u6743\u91cd\u6807\u51c6\u5316\u7684 2D \u5377\u79ef\u5c42",
|
||||
"<h1>2D Convolution Layer with Weight Standardization</h1>\n<p>This is an implementation of a 2 dimensional convolution layer with <a href=\"./index.html\">Weight Standardization</a></p>\n": "<h1>\u5177\u6709\u6743\u91cd\u6807\u51c6\u5316\u7684 2D \u5377\u79ef\u5c42</h1>\n<p>\u8fd9\u662f\u5177\u6709\u6743<a href=\"./index.html\">\u91cd\u6807\u51c6\u5316\u7684</a>\u4e8c\u7ef4\u5377\u79ef\u5c42\u7684\u5b9e\u73b0</p>\n",
|
||||
"<h2>2D Convolution Layer</h2>\n<p>This extends the standard 2D Convolution layer and standardize the weights before the convolution step.</p>\n": "<h2>2D \u5377\u79ef\u5c42</h2>\n<p>\u8fd9\u5c06\u6269\u5c55\u6807\u51c6 2D \u5377\u79ef\u5c42\uff0c\u5e76\u5728\u5377\u79ef\u6b65\u9aa4\u4e4b\u524d\u6807\u51c6\u5316\u6743\u91cd\u3002</p>\n",
|
||||
"<p> A simple test to verify the tensor sizes</p>\n": "<p>\u9a8c\u8bc1\u5f20\u91cf\u5927\u5c0f\u7684\u7b80\u5355\u6d4b\u8bd5</p>\n",
|
||||
"A PyTorch implementation/tutorial of a 2D Convolution Layer with Weight Standardization.": "\u5177\u6709\u6743\u91cd\u6807\u51c6\u5316\u7684\u4e8c\u7ef4\u5377\u79ef\u5c42\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment to try Weight Standardization and Batch-Channel Normalization</h1>\n": "<h1>\u91cd\u307f\u6a19\u6e96\u5316\u3068\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u3092\u8a66\u3059\u305f\u3081\u306eCIFAR10\u5b9f\u9a13</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u30e2\u30c7\u30eb\u4f5c\u6210</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>CIFAR-10 \u5206\u985e\u7528\u306e VGG \u30e2\u30c7\u30eb</h3>\n<p><a href=\"../../experiments/cifar10.html\">\u3053\u308c\u306f\u4e00\u822c\u7684\u306a VGG \u30b9\u30bf\u30a4\u30eb\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306b\u7531\u6765\u3057\u307e\u3059</a>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u69cb\u6210\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5b9f\u9a13\u3092\u958b\u59cb\u3057\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3092\u5b9f\u884c\u3057\u307e\u3059</p>\n",
|
||||
"CIFAR10 Experiment to try Weight Standardization and Batch-Channel Normalization": "\u91cd\u307f\u6a19\u6e96\u5316\u3068\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u3092\u8a66\u3059\u305f\u3081\u306eCIFAR10\u5b9f\u9a13",
|
||||
"This trains is a VGG net that uses weight standardization and batch-channel normalization to classify CIFAR10 images.": "\u3053\u306e\u30c8\u30ec\u30a4\u30f3\u306f\u3001\u91cd\u307f\u6a19\u6e96\u5316\u3068\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u3092\u4f7f\u7528\u3057\u3066 CIFAR10 \u753b\u50cf\u3092\u5206\u985e\u3059\u308b VGG \u30cd\u30c3\u30c8\u3067\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment to try Weight Standardization and Batch-Channel Normalization</h1>\n": "<h1>\u0db6\u0dbb\u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dc4 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d8b\u0dad\u0dca\u0dc3\u0dcf\u0dc4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf CIFAR10 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>CIFA-10\u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf VGG \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h3>\n<p>\u0db8\u0dd9\u0dba <a href=\"../../experiments/cifar10.html\">\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba VGG \u0dc0\u0dd2\u0dbd\u0dcf\u0dc3\u0dd2\u0dad\u0dcf\u0dc0\u0dda \u0d9c\u0dd8\u0dc4 \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab \u0dc1\u0dd2\u0dbd\u0dca\u0db4\u0dba\u0dd9\u0db1\u0dca</a>\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dd3. </p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dbd\u0dd6\u0db4\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"CIFAR10 Experiment to try Weight Standardization and Batch-Channel Normalization": "\u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dc4 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d8b\u0dad\u0dca\u0dc3\u0dcf\u0dc4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf CIFAR10 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This trains is a VGG net that uses weight standardization and batch-channel normalization to classify CIFAR10 images.": "\u0db8\u0dd9\u0db8 \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba CIFAR10 \u0dbb\u0dd6\u0db4 \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dc4 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 VGG \u0daf\u0dd0\u0dbd\u0d9a\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<h1>CIFAR10 Experiment to try Weight Standardization and Batch-Channel Normalization</h1>\n": "<h1>CIFAR10 \u8bd5\u9a8c\uff0c\u5c1d\u8bd5\u6743\u91cd\u6807\u51c6\u5316\u548c\u6279\u6b21\u901a\u9053\u89c4\u8303\u5316</h1>\n",
|
||||
"<h3>Create model</h3>\n": "<h3>\u521b\u5efa\u6a21\u578b</h3>\n",
|
||||
"<h3>VGG model for CIFAR-10 classification</h3>\n<p>This derives from the <a href=\"../../experiments/cifar10.html\">generic VGG style architecture</a>.</p>\n": "<h3>\u7528\u4e8e CIFAR-10 \u5206\u7c7b\u7684 VGG \u6a21\u578b</h3>\n<p>\u8fd9\u6e90\u4e8e<a href=\"../../experiments/cifar10.html\">\u901a\u7528\u7684 VGG \u98ce\u683c\u67b6\u6784</a>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Start the experiment and run the training loop </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c\u5e76\u8fd0\u884c\u8bad\u7ec3\u5faa\u73af</p>\n",
|
||||
"CIFAR10 Experiment to try Weight Standardization and Batch-Channel Normalization": "CIFAR10 \u8bd5\u9a8c\uff0c\u5c1d\u8bd5\u6743\u91cd\u6807\u51c6\u5316\u548c\u6279\u6b21\u901a\u9053\u89c4\u8303\u5316",
|
||||
"This trains is a VGG net that uses weight standardization and batch-channel normalization to classify CIFAR10 images.": "\u8be5\u5217\u8f66\u662f\u4e00\u4e2a VGG \u7f51\uff0c\u5b83\u4f7f\u7528\u6743\u91cd\u6807\u51c6\u5316\u548c\u6279\u91cf\u901a\u9053\u5f52\u4e00\u5316\u5bf9 CIFAR10 \u56fe\u50cf\u8fdb\u884c\u5206\u7c7b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/weight_standardization/index.html\">Weight Standardization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Weight Standardization from the paper <a href=\"https://arxiv.org/abs/1903.10520\">Micro-Batch Training with Batch-Channel Normalization and Weight Standardization</a>. We also have an <a href=\"https://nn.labml.ai/normalization/batch_channel_norm/index.html\">annotated implementation of Batch-Channel Normalization</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/weight_standardization/index.html\">\u91cd\u91cf\u6a19\u6e96\u5316</a></h1>\n<p>\u3053\u308c\u306f\u3001\u8ad6\u6587\u300c<a href=\"https://arxiv.org/abs/1903.10520\">\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u3068\u4f53\u91cd\u6a19\u6e96\u5316\u306b\u3088\u308b\u30de\u30a4\u30af\u30ed\u30d0\u30c3\u30c1\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u300d<a href=\"https://pytorch.org\">\u304b\u3089\u5f15\u7528\u3057\u305f\u91cd\u307f\u6a19\u6e96\u5316\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002\u307e\u305f\u3001<a href=\"https://nn.labml.ai/normalization/batch_channel_norm/index.html\">\u30d0\u30c3\u30c1\u30c1\u30e3\u30cd\u30eb\u6b63\u898f\u5316\u306e\u6ce8\u91c8\u4ed8\u304d\u5b9f\u88c5\u3082\u3042\u308a\u307e\u3059\u3002</a></p>\n",
|
||||
"Weight Standardization": "\u91cd\u91cf\u6a19\u6e96\u5316"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/weight_standardization/index.html\">Weight Standardization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Weight Standardization from the paper <a href=\"https://arxiv.org/abs/1903.10520\">Micro-Batch Training with Batch-Channel Normalization and Weight Standardization</a>. We also have an <a href=\"https://nn.labml.ai/normalization/batch_channel_norm/index.html\">annotated implementation of Batch-Channel Normalization</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/weight_standardization/index.html\">\u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba</a></h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2 \u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dbd\u0dd2\u0db1\u0dca <a href=\"https://arxiv.org/abs/1903.10520\">\u0d9a\u0dca\u0dc2\u0dd4\u0daf\u0dca\u0dbb \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0dc4 \u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f</a> . <a href=\"https://nn.labml.ai/normalization/batch_channel_norm/index.html\">\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca-\u0db1\u0dcf\u0dbd\u0dd2\u0d9a\u0dcf \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc0\u0dd2\u0da0\u0dd2\u0dad\u0dca\u0dbb\u0dc0\u0dad\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a</a>\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dca \u0daf \u0d85\u0db4 \u0dc3\u0dad\u0dd4\u0dc0 \u0d87\u0dad. </p>\n",
|
||||
"Weight Standardization": "\u0db6\u0dbb \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/normalization/weight_standardization/index.html\">Weight Standardization</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of Weight Standardization from the paper <a href=\"https://arxiv.org/abs/1903.10520\">Micro-Batch Training with Batch-Channel Normalization and Weight Standardization</a>. We also have an <a href=\"https://nn.labml.ai/normalization/batch_channel_norm/index.html\">annotated implementation of Batch-Channel Normalization</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/normalization/weight_standardization/index.html\">\u91cd\u91cf\u6807\u51c6\u5316</a></h1>\n<p>\u8fd9\u662f <a href=\"https://pytorch.org\">PyTorch</a> \u5728\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/1903.10520\">\u4f7f\u7528\u6279\u6b21\u901a\u9053\u6807\u51c6\u5316\u548c\u6743\u91cd\u6807\u51c6\u5316\u7684\u5fae\u6279\u91cf\u8bad\u7ec3\u300b\u4e2d\u5b9e\u73b0\u7684\u91cd\u91cf\u6807\u51c6\u5316</a>\u3002\u6211\u4eec\u8fd8\u6709\u4e00\u4e2a<a href=\"https://nn.labml.ai/normalization/batch_channel_norm/index.html\">\u5e26\u6ce8\u91ca\u7684\u6279\u5904\u7406\u4fe1\u9053\u89c4\u8303\u5316\u5b9e\u73b0</a>\u3002</p>\n",
|
||||
"Weight Standardization": "\u91cd\u91cf\u6807\u51c6\u5316"
|
||||
}
|
||||
Reference in New Issue
Block a user