chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,78 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Masked Language Model (MLM)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">Masked Language Model</a>.</p>\n": "<h1><a href=\"index.html\">\u30de\u30b9\u30af\u8a00\u8a9e\u30e2\u30c7\u30eb (MLM</a>) \u5b9f\u9a13</h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f\u3001\u30de\u30b9\u30af\u8a00\u8a9e\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304dPyTorch\u5b9f\u9a13\u3067\u3059\u3002</a></p>\n",
|
||||
"<h1>Transformer based model for MLM</h1>\n": "<h1>MLM \u7528\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30d9\u30fc\u30b9\u30e2\u30c7\u30eb</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a> because it has the data pipeline implementations that we reuse here. We have implemented a custom training step form MLM.</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p><a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a>\u3053\u308c\u304c\u7d99\u627f\u3055\u308c\u3066\u3044\u308b\u306e\u306f\u3001\u3053\u3053\u3067\u518d\u5229\u7528\u3059\u308b\u30c7\u30fc\u30bf\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u306e\u5b9f\u88c5\u304c\u3042\u308b\u304b\u3089\u3067\u3059\u3002MLM\u304b\u3089\u30ab\u30b9\u30bf\u30e0\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b9\u30c6\u30c3\u30d7\u3092\u5b9f\u88c5\u3057\u307e\u3057\u305f</p>\u3002\n",
|
||||
"<h3>Initialization</h3>\n": "<h3>\u521d\u671f\u5316</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\u5b9a\u671f\u7684\u306b\u30b5\u30f3\u30d7\u30eb\u3092\u751f\u6210\u3059\u308b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u6a5f\u80fd</h3>\n",
|
||||
"<h3>Training or validation step</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u307e\u305f\u306f\u691c\u8a3c\u30b9\u30c6\u30c3\u30d7</h3>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u5909\u5727\u5668\u69cb\u6210</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create classification model</p>\n": "<p>\u5206\u985e\u30e2\u30c7\u30eb\u306e\u4f5c\u6210</p>\n",
|
||||
"<p> Number of tokens including <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u304a\u3088\u3073\u3092\u542b\u3080\u30c8\u30fc\u30af\u30f3\u306e\u6570 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p><a href=\"index.html\">Masked Language Model (MLM) class</a> to generate the mask </p>\n": "<p><a href=\"index.html\">\u30de\u30b9\u30af\u3092\u751f\u6210\u3059\u308b\u30de\u30b9\u30af\u8a00\u8a9e\u30e2\u30c7\u30eb (MLM) \u30af\u30e9\u30b9</a></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> token </p>\n": "<p><span translate=no>_^_0_^_</span>\u30c8\u30fc\u30af\u30f3</p>\n",
|
||||
"<p>Accuracy metric (ignore the labels equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u7cbe\u5ea6\u6307\u6a19 (\u3068\u7b49\u3057\u3044\u30e9\u30d9\u30eb\u306f\u7121\u8996\u3057\u3066\u304f\u3060\u3055\u3044<span translate=no>_^_0_^_</span>)</p>\n",
|
||||
"<p>Add the prompts one by one </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u3092 1 \u3064\u305a\u3064\u8ffd\u52a0\u3057\u307e\u3059</p>\n",
|
||||
"<p>Add to the tensor </p>\n": "<p>\u30c6\u30f3\u30bd\u30eb\u306b\u8ffd\u52a0</p>\n",
|
||||
"<p>Batch size </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u7cbe\u5ea6\u306e\u8a08\u7b97\u3068\u8a18\u9332</p>\n",
|
||||
"<p>Calculate and log the loss </p>\n": "<p>\u640d\u5931\u306e\u8a08\u7b97\u3068\u8a18\u9332</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u52fe\u914d\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30af\u30ea\u30a2</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u30af\u30ea\u30c3\u30d7\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Collect output from printing </p>\n": "<p>\u5370\u5237\u304b\u3089\u306e\u51fa\u529b\u3092\u53ce\u96c6</p>\n",
|
||||
"<p>Correct prediction </p>\n": "<p>\u6b63\u3057\u3044\u4e88\u6e2c</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Cross entropy loss (ignore the labels equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u30af\u30ed\u30b9\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u640d\u5931 (\u3068\u7b49\u3057\u3044\u30e9\u30d9\u30eb\u306f\u7121\u8996\u3057\u3066\u304f\u3060\u3055\u3044) <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Empty tensor for data filled with <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u304c\u5165\u529b\u3055\u308c\u305f\u30c7\u30fc\u30bf\u306e\u30c6\u30f3\u30bd\u30eb\u3092\u7a7a\u306b\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>For each token </p>\n": "<p>\u5404\u30c8\u30fc\u30af\u30f3\u306b\u3064\u3044\u3066</p>\n",
|
||||
"<p>Get masked input and labels </p>\n": "<p>\u30de\u30b9\u30af\u3055\u308c\u305f\u5165\u529b\u3068\u30e9\u30d9\u30eb\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get model outputs </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get model outputs. It's returning a tuple for states when using RNNs. This is not implemented yet. </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u53d6\u5f97\u3057\u307e\u3059\u3002RNN \u3092\u4f7f\u7528\u3059\u308b\u5834\u5408\u306f\u30b9\u30c6\u30fc\u30c8\u306e\u30bf\u30d7\u30eb\u3092\u8fd4\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u307e\u3060\u5b9f\u88c5\u3055\u308c\u3066\u3044\u307e\u305b\u3093\u3002</p>\n",
|
||||
"<p>Get the masked input and labels </p>\n": "<p>\u30de\u30b9\u30af\u3055\u308c\u305f\u5165\u529b\u3068\u30e9\u30d9\u30eb\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the prediction </p>\n": "<p>\u4e88\u6e2c\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u306b\u3088\u308b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get token indexes </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>If it's a printable character </p>\n": "<p>\u5370\u5237\u53ef\u80fd\u306a\u6587\u5b57\u306e\u5834\u5408</p>\n",
|
||||
"<p>If it's not a printable character </p>\n": "<p>\u5370\u5237\u53ef\u80fd\u306a\u6587\u5b57\u3067\u306a\u3044\u5834\u5408</p>\n",
|
||||
"<p>If the label is <span translate=no>_^_0_^_</span> (unmasked) print the original. </p>\n": "<p>\u30e9\u30d9\u30eb\u304c <span translate=no>_^_0_^_</span> (\u30de\u30b9\u30af\u3055\u308c\u3066\u3044\u306a\u3044) \u5834\u5408\u306f\u3001\u30aa\u30ea\u30b8\u30ca\u30eb\u3092\u5370\u5237\u3057\u3066\u304f\u3060\u3055\u3044\u3002</p>\n",
|
||||
"<p>If the label is not <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30e9\u30d9\u30eb\u304c\u305d\u3046\u3067\u306a\u3044\u5834\u5408 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Incorrect prediction </p>\n": "<p>\u4e88\u6e2c\u304c\u9593\u9055\u3063\u3066\u3044\u308b</p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u5404\u30a8\u30dd\u30c3\u30af\u306e\u6700\u5f8c\u306e\u30d0\u30c3\u30c1\u3067\u30e2\u30c7\u30eb\u30d1\u30e9\u30e1\u30fc\u30bf\u3068\u52fe\u914d\u3092\u8a18\u9332\u3057\u307e\u3059</p>\n",
|
||||
"<p>Logits for the output </p>\n": "<p>\u51fa\u529b\u7528\u306e\u30ed\u30b8\u30c3\u30c8</p>\n",
|
||||
"<p>MLM model </p>\n": "<p>MLM \u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>Move the input to the device </p>\n": "<p>\u5165\u529b\u3092\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Move the tensor to current device </p>\n": "<p>\u30c6\u30f3\u30bd\u30eb\u3092\u73fe\u5728\u306e\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Number of tokens </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u6570</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Print </p>\n": "<p>\u30d7\u30ea\u30f3\u30c8</p>\n",
|
||||
"<p>Print the samples generated </p>\n": "<p>\u751f\u6210\u3055\u308c\u305f\u30b5\u30f3\u30d7\u30eb\u3092\u5370\u5237</p>\n",
|
||||
"<p>Probability of masking a token </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u3092\u30de\u30b9\u30ad\u30f3\u30b0\u3059\u308b\u78ba\u7387</p>\n",
|
||||
"<p>Probability of replacing the mask with a random token </p>\n": "<p>\u30de\u30b9\u30af\u3092\u30e9\u30f3\u30c0\u30e0\u30c8\u30fc\u30af\u30f3\u306b\u7f6e\u304d\u63db\u3048\u308b\u78ba\u7387</p>\n",
|
||||
"<p>Probability of replacing the mask with original token </p>\n": "<p>\u30de\u30b9\u30af\u3092\u5143\u306e\u30c8\u30fc\u30af\u30f3\u3068\u4ea4\u63db\u3059\u308b\u78ba\u7387</p>\n",
|
||||
"<p>Prompt to sample </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3092\u4fc3\u3059</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u7d50\u679c\u3092\u8fd4\u3057\u307e\u3059\uff08\u30c8\u30ec\u30fc\u30ca\u30fc\u306fRNN\u3067\u3082\u4f7f\u7528\u3055\u308c\u308b\u305f\u3081\u30012\u756a\u76ee\u306e\u5024\u306f\u72b6\u614b\u7528\u3067\u3059\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u8ffd\u8de1\u3057\u305f\u30e1\u30c8\u30ea\u30af\u30b9\u3092\u4fdd\u5b58\u3059\u308b</p>\n",
|
||||
"<p>Sequence length of <span translate=no>_^_0_^_</span>. We use a short sequence length to train faster. Otherwise it takes forever to train. </p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u306f <span translate=no>_^_0_^_</span>\u77ed\u3044\u30b7\u30fc\u30b1\u30f3\u30b9\u9577\u3092\u4f7f\u7528\u3057\u3066\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u9ad8\u901f\u5316\u3057\u307e\u3059\u3002\u305d\u3046\u3057\u306a\u3044\u3068\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306b\u6642\u9593\u304c\u304b\u304b\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u6700\u9069\u5316\u306e\u4e00\u6b69\u3092\u8e0f\u307f\u51fa\u3059</p>\n",
|
||||
"<p>Tokens that shouldn't be masked </p>\n": "<p>\u30de\u30b9\u30af\u3057\u3066\u306f\u3044\u3051\u306a\u3044\u30c8\u30fc\u30af\u30f3</p>\n",
|
||||
"<p>Train for 1024 epochs. </p>\n": "<p>1024 \u30a8\u30dd\u30c3\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u884c\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u5909\u5727\u5668\u69cb\u6210 (\u30c7\u30d5\u30a9\u30eb\u30c8\u3068\u540c\u3058)</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u30c8\u30e9\u30f3\u30b9\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc</p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30e2\u30fc\u30c9\u6642\u306b\u30b0\u30ed\u30fc\u30d0\u30eb\u30b9\u30c6\u30c3\u30d7 (\u51e6\u7406\u3055\u308c\u305f\u30c8\u30fc\u30af\u30f3\u306e\u6570) \u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u30ad\u30e3\u30d7\u30c1\u30e3\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><a href=\"../models.html#Encoder\">\u5909\u5727\u5668\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</a></li>\n<li><span translate=no>_^_1_^_</span><a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u306f\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb\u3067\u3059 (\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u4ed8\u304d)</a></li>\n</ul><li><span translate=no>_^_2_^_</span><a href=\"../models.html#Generator\">\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210\u3059\u308b\u6700\u5f8c\u306e\u5b8c\u5168\u63a5\u7d9a\u5c64\u3067\u3059</a>\u3002</li>\n",
|
||||
"Masked Language Model Experiment": "\u4eee\u9762\u8a00\u8a9e\u30e2\u30c7\u30eb\u5b9f\u9a13",
|
||||
"This experiment trains Masked Language Model (MLM) on Tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001Tiny Shakespeare\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067\u30de\u30b9\u30af\u8a00\u8a9e\u30e2\u30c7\u30eb (MLM) \u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Masked Language Model (MLM)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">Masked Language Model</a>.</p>\n": "<h1><a href=\"index.html\">\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd6\u0da9\u0dca \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba (\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0d91\u0db8\u0dca)</a> \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">\u0dc0\u0dd9\u0dc3\u0dca \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0dbb\u0dca\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n",
|
||||
"<h1>Transformer based model for MLM</h1>\n": "<h1>\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0d91\u0db8\u0dca\u0dc3\u0db3\u0dc4\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a> because it has the data pipeline implementations that we reuse here. We have implemented a custom training step form MLM.</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db8\u0dd9\u0dba\u0d8b\u0dbb\u0dd4\u0db8 \u0dc0\u0db1\u0dca\u0db1\u0dda \u0d85\u0db4 \u0db8\u0dd9\u0dc4\u0dd2 \u0db1\u0dd0\u0dc0\u0dad \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0daf\u0dad\u0dca\u0dad \u0db1\u0dbd \u0db8\u0dcf\u0dbb\u0dca\u0d9c \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca \u0d87\u0dad\u0dd2 <a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a> \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dd2. \u0d85\u0db4\u0dd2 \u0d85\u0db7\u0dd2\u0dbb\u0dd4\u0da0\u0dd2 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0d9a\u0dca MLM \u0db4\u0ddd\u0dbb\u0db8\u0dba\u0d9a\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb \u0d87\u0dad\u0dca\u0dad\u0dd9\u0db8\u0dd4. </p>\n",
|
||||
"<h3>Initialization</h3>\n": "<h3>\u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a\u0d9a\u0dbb\u0dab\u0dba</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0dc0\u0dbb\u0dd2\u0db1\u0dca \u0dc0\u0dbb \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba</h3>\n",
|
||||
"<h3>Training or validation step</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0dc4\u0ddd \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dd2\u0dba\u0dc0\u0dbb</h3>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create classification model</p>\n": "<p> \u0dc0\u0dbb\u0dca\u0d9c\u0dd3\u0d9a\u0dbb\u0dab\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p> Number of tokens including <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span></p>\n": "<p> \u0d87\u0dad\u0dd4\u0dc5\u0dd4\u0dc0 <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p><a href=\"index.html\">Masked Language Model (MLM) class</a> to generate the mask </p>\n": "<p><a href=\"index.html\">\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba (\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0d91\u0db8\u0dca) \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba</a> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> token </p>\n": "<p><span translate=no>_^_0_^_</span> \u0da7\u0ddd\u0d9a\u0db1\u0dba </p>\n",
|
||||
"<p>Accuracy metric (ignore the labels equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0db8\u0dd9\u0da7\u0dca\u0dbb\u0dd2\u0d9a\u0dca (\u0dc3\u0db8\u0dcf\u0db1 \u0dbd\u0dda\u0db6\u0dbd\u0dca \u0db1\u0ddc\u0dc3\u0dbd\u0d9a\u0dcf \u0dc4\u0dbb\u0dd2\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span>) </p>\n",
|
||||
"<p>Add the prompts one by one </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd3\u0db8\u0dca\u0d91\u0d9a\u0dd2\u0db1\u0dca \u0d91\u0d9a \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add to the tensor </p>\n": "<p>\u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dba\u0da7\u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Batch size </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba </p>\n",
|
||||
"<p>Calculate and log the loss </p>\n": "<p>\u0d85\u0dbd\u0dcf\u0db7\u0dba\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d89\u0dc0\u0dad\u0dca </p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dca\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a </p>\n",
|
||||
"<p>Collect output from printing </p>\n": "<p>\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba\u0dd9\u0db1\u0dca\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Correct prediction </p>\n": "<p>\u0db1\u0dd2\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Cross entropy loss (ignore the labels equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u0dc4\u0dbb\u0dc3\u0dca\u0d91\u0db1\u0dca\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd2\u0dba \u0db1\u0dd0\u0dad\u0dd2\u0dc0\u0dd3\u0db8 (\u0dc3\u0db8\u0dcf\u0db1 \u0dbd\u0dda\u0db6\u0dbd\u0dca \u0db1\u0ddc\u0dc3\u0dbd\u0d9a\u0dcf \u0dc4\u0dbb\u0dd2\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span>) </p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Empty tensor for data filled with <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u0db4\u0dd4\u0dbb\u0dc0\u0dcf\u0d87\u0dad\u0dd2 \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db3\u0dc4\u0dcf \u0dc4\u0dd2\u0dc3\u0dca \u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dca <span translate=no>_^_0_^_</span>. </p>\n",
|
||||
"<p>For each token </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0da7\u0ddd\u0d9a\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf </p>\n",
|
||||
"<p>Get masked input and labels </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0d86\u0daf\u0dcf\u0db1 \u0dc3\u0dc4 \u0dbd\u0dda\u0db6\u0dbd \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get model outputs </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get model outputs. It's returning a tuple for states when using RNNs. This is not implemented yet. </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1. \u0d86\u0dbb\u0dca\u0d91\u0db1\u0dca\u0d91\u0dc3\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0dc0\u0dd2\u0da7 \u0d91\u0dba \u0db4\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dad \u0dc3\u0db3\u0dc4\u0dcf \u0da7\u0dd6\u0dbd\u0dca \u0d91\u0d9a\u0d9a\u0dca \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0dba\u0dd2. \u0db8\u0dd9\u0dba \u0dad\u0dc0\u0db8\u0dad\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0db1\u0ddc\u0dc0\u0dda. </p>\n",
|
||||
"<p>Get the masked input and labels </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd6\u0da9\u0dca\u0d86\u0daf\u0dcf\u0db1\u0dba \u0dc3\u0dc4 \u0dbd\u0dda\u0db6\u0dbd \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the prediction </p>\n": "<p>\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba\u0d9a\u0dda\u0dad\u0db1 \u0d9a\u0dca\u0dbb\u0db8 \u0dc3\u0db8\u0d9f \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get token indexes </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0daf\u0dbb\u0dca\u0dc1\u0d9a \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>If it's a printable character </p>\n": "<p>\u0d91\u0dba\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da0\u0dbb\u0dd2\u0dad\u0dba\u0d9a\u0dca \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>If it's not a printable character </p>\n": "<p>\u0d91\u0dba\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da0\u0dbb\u0dd2\u0dad\u0dba\u0d9a\u0dca \u0db1\u0ddc\u0dc0\u0dda \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>If the label is <span translate=no>_^_0_^_</span> (unmasked) print the original. </p>\n": "<p>\u0dbd\u0dda\u0db6\u0dbd\u0dba <span translate=no>_^_0_^_</span> (\u0db1\u0ddc\u0d9a\u0dd0\u0da9\u0dd6) \u0db1\u0db8\u0dca \u0db8\u0dd4\u0dbd\u0dca \u0db4\u0dd2\u0da7\u0db4\u0dad \u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>If the label is not <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dbd\u0dda\u0db6\u0dbd\u0dba\u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db1\u0db8\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Incorrect prediction </p>\n": "<p>\u0dc0\u0dd0\u0dbb\u0daf\u0dd2\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba </p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u0dc3\u0dd1\u0db8\u0dba\u0dd4\u0d9c\u0dbd\u0dba\u0d9a\u0db8 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0dc4 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Logits for the output </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0dbd\u0ddc\u0d9c\u0dd2\u0db1\u0dca \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>MLM model </p>\n": "<p>\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0d91\u0db8\u0dca\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>Move the input to the device </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0dba\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0da7 \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move the tensor to current device </p>\n": "<p>\u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dba\u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca \u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0da7 \u0d9c\u0dd9\u0db1 \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Number of tokens </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Print </p>\n": "<p>\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba </p>\n",
|
||||
"<p>Print the samples generated </p>\n": "<p>\u0da2\u0db1\u0db1\u0dba\u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Probability of masking a token </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0d9a\u0dca\u0d86\u0dc0\u0dbb\u0dab \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Probability of replacing the mask with a random token </p>\n": "<p>\u0d85\u0dc4\u0db9\u0dd4\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Probability of replacing the mask with original token </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0db8\u0dd4\u0dbd\u0dca \u0da7\u0ddd\u0d9a\u0db1\u0dba \u0dc3\u0db8\u0d9f \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db4\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Prompt to sample </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba\u0dc0\u0dd9\u0dad \u0dc0\u0dd2\u0db8\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0dcf\u0db7\u0db4\u0dca\u0dbb\u0dad\u0dd2 results \u0dbd (\u0daf\u0dd9\u0dc0\u0db1 \u0d85\u0d9c\u0dba \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dda, \u0db8\u0db1\u0dca\u0daf \u0d85\u0db4\u0d9c\u0dda \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4 RNs \u0dc3\u0db8\u0d9f \u0daf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2) </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u0dbd\u0dd4\u0dc4\u0dd4\u0db6\u0dd0\u0db3\u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Sequence length of <span translate=no>_^_0_^_</span>. We use a short sequence length to train faster. Otherwise it takes forever to train. </p>\n": "<p>\u0d85\u0db1\u0dd4\u0db4\u0dd2\u0dc5\u0dd2\u0dc0\u0dd9\u0dbd\u0daf\u0dd2\u0d9c <span translate=no>_^_0_^_</span>. \u0dc0\u0dda\u0d9c\u0dba\u0dd9\u0db1\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 \u0d9a\u0dd9\u0da7\u0dd2 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0daf\u0dd2\u0d9c\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4. \u0d91\u0dc3\u0dda \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2\u0db1\u0db8\u0dca \u0d91\u0dba \u0dc3\u0daf\u0dc4\u0da7\u0db8 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d9c\u0dad \u0dc0\u0dda. </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Tokens that shouldn't be masked </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd6\u0dc3\u0dca\u0db1\u0ddc\u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0da7\u0ddd\u0d9a\u0db1 </p>\n",
|
||||
"<p>Train for 1024 epochs. </p>\n": "<p>1024\u0d91\u0db4\u0ddc\u0da0\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba. </p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca (\u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dbd\u0dd9\u0dc3) </p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db4\u0dca\u0dbb\u0d9a\u0dcf\u0dbb\u0dba\u0dda\u0daf\u0dd3 \u0d9c\u0ddd\u0dbd\u0dd3\u0dba \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 (\u0dc3\u0dd0\u0d9a\u0dc3\u0dd6 \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1) </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d9c\u0dca\u0dbb\u0dc4\u0dab\u0dba \u0d9a\u0dbb \u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca <a href=\"../models.html#Encoder\">\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba</a> </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0da7\u0ddd\u0d9a\u0db1\u0dca <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba (\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dda\u0dad\u0dd3\u0d9a\u0dbb\u0dab \u0dc3\u0db8\u0d9f)</a> </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1 <a href=\"../models.html#Generator\">\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dba\u0dd2</a> . </li></ul>\n",
|
||||
"Masked Language Model Experiment": "\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd6\u0da9\u0dca \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains Masked Language Model (MLM) on Tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba\u0dda \u0dc0\u0dd9\u0dc3\u0dca \u0db7\u0dcf\u0dc2\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba (\u0d91\u0db8\u0dca\u0d91\u0dbd\u0dca\u0d91\u0db8\u0dca) \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
{
|
||||
"<h1><a href=\"index.html\">Masked Language Model (MLM)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">Masked Language Model</a>.</p>\n": "<h1><a href=\"index.html\">\u63a9\u7801\u8bed\u8a00\u6a21\u578b (MLM)</a> \u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u4e2a\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3\u4e00\u4e2a<a href=\"index.html\">\u8499\u7248\u8bed\u8a00\u6a21\u578b</a>\u3002</p>\n",
|
||||
"<h1>Transformer based model for MLM</h1>\n": "<h1>\u57fa\u4e8e\u53d8\u538b\u5668\u7684\u4f20\u9500\u6a21\u578b</h1>\n",
|
||||
"<h2>Configurations</h2>\n<p>This inherits from <a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a> because it has the data pipeline implementations that we reuse here. We have implemented a custom training step form MLM.</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u8fd9\u7ee7\u627f\u81ea\uff0c<a href=\"../../experiments/nlp_autoregression.html\"><span translate=no>_^_0_^_</span></a>\u56e0\u4e3a\u5b83\u6709\u6211\u4eec\u5728\u8fd9\u91cc\u91cd\u7528\u7684\u6570\u636e\u7ba1\u9053\u5b9e\u73b0\u3002\u6211\u4eec\u5df2\u7ecf\u5b9e\u65bd\u4e86 MLM \u7684\u81ea\u5b9a\u4e49\u8bad\u7ec3\u6b65\u9aa4\u3002</p>\n",
|
||||
"<h3>Initialization</h3>\n": "<h3>\u521d\u59cb\u5316</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u91c7\u6837\u529f\u80fd\u53ef\u5728\u8bad\u7ec3\u65f6\u5b9a\u671f\u751f\u6210\u6837\u672c</h3>\n",
|
||||
"<h3>Training or validation step</h3>\n": "<h3>\u57f9\u8bad\u6216\u9a8c\u8bc1\u6b65\u9aa4</h3>\n",
|
||||
"<h3>Transformer configurations</h3>\n": "<h3>\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create classification model</p>\n": "<p>\u521b\u5efa\u5206\u7c7b\u6a21\u578b</p>\n",
|
||||
"<p> Number of tokens including <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span></p>\n": "<p>\u5305\u62ec<span translate=no>_^_0_^_</span>\u548c\u5728\u5185\u7684\u4ee3\u5e01\u6570\u91cf<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p><a href=\"index.html\">Masked Language Model (MLM) class</a> to generate the mask </p>\n": "<p>\u7528\u4e8e\u751f\u6210<a href=\"index.html\">\u63a9\u7801\u7684\u63a9\u7801\u8bed\u8a00\u6a21\u578b (MLM) \u7c7b</a></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> token </p>\n": "<p><span translate=no>_^_0_^_</span>\u4ee4\u724c</p>\n",
|
||||
"<p>Accuracy metric (ignore the labels equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u7cbe\u5ea6\u5ea6\u91cf\u5ea6\uff08\u5ffd\u7565\u7b49\u4e8e\u7684\u6807\u7b7e<span translate=no>_^_0_^_</span>\uff09</p>\n",
|
||||
"<p>Add the prompts one by one </p>\n": "<p>\u9010\u4e2a\u6dfb\u52a0\u63d0\u793a</p>\n",
|
||||
"<p>Add to the tensor </p>\n": "<p>\u6dfb\u52a0\u5230\u5f20\u91cf\u4e2d</p>\n",
|
||||
"<p>Batch size </p>\n": "<p>\u6279\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u8ba1\u7b97\u548c\u8bb0\u5f55\u7cbe\u5ea6</p>\n",
|
||||
"<p>Calculate and log the loss </p>\n": "<p>\u8ba1\u7b97\u5e76\u8bb0\u5f55\u635f\u5931</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u6e05\u9664\u6e10\u53d8</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u526a\u8f91\u6e10\u53d8</p>\n",
|
||||
"<p>Collect output from printing </p>\n": "<p>\u4ece\u6253\u5370\u4e2d\u6536\u96c6\u8f93\u51fa</p>\n",
|
||||
"<p>Correct prediction </p>\n": "<p>\u6b63\u786e\u7684\u9884\u6d4b</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Cross entropy loss (ignore the labels equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u4ea4\u53c9\u71b5\u635f\u5931\uff08\u5ffd\u7565\u7b49\u4e8e\u7684\u6807\u7b7e<span translate=no>_^_0_^_</span>\uff09</p>\n",
|
||||
"<p>Embedding size </p>\n": "<p>\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Empty tensor for data filled with <span translate=no>_^_0_^_</span>. </p>\n": "<p>\u586b\u5145\u7684\u6570\u636e\u4e3a\u7a7a\u5f20\u91cf<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p>For each token </p>\n": "<p>\u5bf9\u4e8e\u6bcf\u4e2a\u4ee3\u5e01</p>\n",
|
||||
"<p>Get masked input and labels </p>\n": "<p>\u83b7\u53d6\u5c4f\u853d\u7684\u8f93\u5165\u548c\u6807\u7b7e</p>\n",
|
||||
"<p>Get model outputs </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>Get model outputs. It's returning a tuple for states when using RNNs. This is not implemented yet. </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u8f93\u51fa\u3002\u5b83\u5728\u4f7f\u7528 RNN \u65f6\u8fd4\u56de\u72b6\u6001\u7684\u5143\u7ec4\u3002\u8fd9\u5c1a\u672a\u5b9e\u73b0\u3002</p>\n",
|
||||
"<p>Get the masked input and labels </p>\n": "<p>\u83b7\u53d6\u5c4f\u853d\u7684\u8f93\u5165\u548c\u6807\u7b7e</p>\n",
|
||||
"<p>Get the prediction </p>\n": "<p>\u83b7\u53d6\u9884\u6d4b</p>\n",
|
||||
"<p>Get the token embeddings with positional encodings </p>\n": "<p>\u4f7f\u7528\u4f4d\u7f6e\u7f16\u7801\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Get token indexes </p>\n": "<p>\u83b7\u53d6\u4ee3\u5e01\u7d22\u5f15</p>\n",
|
||||
"<p>If it's a printable character </p>\n": "<p>\u5982\u679c\u662f\u53ef\u6253\u5370\u7684\u5b57\u7b26</p>\n",
|
||||
"<p>If it's not a printable character </p>\n": "<p>\u5982\u679c\u5b83\u4e0d\u662f\u53ef\u6253\u5370\u7684\u5b57\u7b26</p>\n",
|
||||
"<p>If the label is <span translate=no>_^_0_^_</span> (unmasked) print the original. </p>\n": "<p>\u5982\u679c\u6807\u7b7e\u662f<span translate=no>_^_0_^_</span>\uff08\u672a\u906e\u7f69\uff09\uff0c\u8bf7\u6253\u5370\u539f\u4ef6\u3002</p>\n",
|
||||
"<p>If the label is not <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u6807\u7b7e\u4e0d\u662f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Incorrect prediction </p>\n": "<p>\u9884\u6d4b\u4e0d\u6b63\u786e</p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u8bb0\u5f55\u6bcf\u4e2a\u7eaa\u5143\u6700\u540e\u4e00\u6279\u7684\u6a21\u578b\u53c2\u6570\u548c\u68af\u5ea6</p>\n",
|
||||
"<p>Logits for the output </p>\n": "<p>\u8f93\u51fa\u7684\u5bf9\u6570</p>\n",
|
||||
"<p>MLM model </p>\n": "<p>\u4f20\u9500\u6a21\u578b</p>\n",
|
||||
"<p>Move the input to the device </p>\n": "<p>\u5c06\u8f93\u5165\u79fb\u81f3\u8bbe\u5907</p>\n",
|
||||
"<p>Move the tensor to current device </p>\n": "<p>\u5c06\u5f20\u91cf\u79fb\u5230\u5f53\u524d\u8bbe\u5907</p>\n",
|
||||
"<p>Number of tokens </p>\n": "<p>\u4ee3\u5e01\u6570\u91cf</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Print </p>\n": "<p>\u6253\u5370</p>\n",
|
||||
"<p>Print the samples generated </p>\n": "<p>\u6253\u5370\u751f\u6210\u7684\u6837\u672c</p>\n",
|
||||
"<p>Probability of masking a token </p>\n": "<p>\u63a9\u76d6\u4ee3\u5e01\u7684\u6982\u7387</p>\n",
|
||||
"<p>Probability of replacing the mask with a random token </p>\n": "<p>\u7528\u968f\u673a\u4ee4\u724c\u66ff\u6362\u63a9\u7801\u7684\u6982\u7387</p>\n",
|
||||
"<p>Probability of replacing the mask with original token </p>\n": "<p>\u7528\u539f\u59cb\u4ee4\u724c\u66ff\u6362\u63a9\u7801\u7684\u6982\u7387</p>\n",
|
||||
"<p>Prompt to sample </p>\n": "<p>\u63d0\u793a\u91c7\u6837</p>\n",
|
||||
"<p>Return results (second value is for state, since our trainer is used with RNNs also) </p>\n": "<p>\u8fd4\u56de\u7ed3\u679c\uff08\u7b2c\u4e8c\u4e2a\u503c\u7528\u4e8e\u72b6\u6001\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u8bad\u7ec3\u5668\u4e5f\u4e0e RNN \u4e00\u8d77\u4f7f\u7528\uff09</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u4fdd\u5b58\u8ddf\u8e2a\u7684\u6307\u6807</p>\n",
|
||||
"<p>Sequence length of <span translate=no>_^_0_^_</span>. We use a short sequence length to train faster. Otherwise it takes forever to train. </p>\n": "\u7684@@ <p>\u5e8f\u5217\u957f\u5ea6<span translate=no>_^_0_^_</span>\u3002\u6211\u4eec\u4f7f\u7528\u8f83\u77ed\u7684\u5e8f\u5217\u957f\u5ea6\u6765\u66f4\u5feb\u5730\u8bad\u7ec3\u3002\u5426\u5219\u8bad\u7ec3\u9700\u8981\u5f88\u957f\u65f6\u95f4\u3002</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u91c7\u53d6\u4f18\u5316\u5668\u6b65\u9aa4</p>\n",
|
||||
"<p>Tokens that shouldn't be masked </p>\n": "<p>\u4e0d\u5e94\u8be5\u88ab\u63a9\u76d6\u7684\u4ee3\u5e01</p>\n",
|
||||
"<p>Train for 1024 epochs. </p>\n": "<p>\u8bad\u7ec3 1024 \u4e2a\u65f6\u4ee3\u3002</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u8bad\u7ec3\u6a21\u578b</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Transformer configurations (same as defaults) </p>\n": "<p>\u53d8\u538b\u5668\u914d\u7f6e\uff08\u4e0e\u9ed8\u8ba4\u503c\u76f8\u540c\uff09</p>\n",
|
||||
"<p>Transformer encoder </p>\n": "<p>\u53d8\u538b\u5668\u7f16\u7801</p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u5728\u8bad\u7ec3\u6a21\u5f0f\u4e0b\u66f4\u65b0\u5168\u5c40\u6b65\u957f\uff08\u5904\u7406\u7684\u4ee4\u724c\u6570\uff09</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u662f\u5426\u6355\u83b7\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer <a href=\"../models.html#Encoder\">Encoder</a> </li>\n<li><span translate=no>_^_1_^_</span> is the token <a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">embedding module (with positional encodings)</a> </li>\n<li><span translate=no>_^_2_^_</span> is the <a href=\"../models.html#Generator\">final fully connected layer</a> that gives the logits.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668<a href=\"../models.html#Encoder\">\u7f16\u7801\u5668</a></li>\n<li><span translate=no>_^_1_^_</span>\u662f\u4ee4\u724c<a href=\"../models.html#EmbeddingsWithLearnedPositionalEncoding\">\u5d4c\u5165\u6a21\u5757\uff08\u5e26\u6709\u4f4d\u7f6e\u7f16\u7801\uff09</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u7ed9<a href=\"../models.html#Generator\">\u51fa logit \u7684\u6700\u540e\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42</a>\u3002</li></ul>\n",
|
||||
"Masked Language Model Experiment": "\u8499\u9762\u8bed\u8a00\u6a21\u578b\u5b9e\u9a8c",
|
||||
"This experiment trains Masked Language Model (MLM) on Tiny Shakespeare dataset.": "\u672c\u5b9e\u9a8c\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u8499\u7248\u8bed\u8a00\u6a21\u578b\uff08MLM\uff09\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user