chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"<h1>Compressive Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a compressive transformer model.</p>\n": "<h1>\u5727\u7e2e\u5909\u5727\u5668\u5b9f\u9a13</h1>\n<p>\u3053\u308c\u306f\u3001\u5727\u7e2e\u30c8\u30e9\u30f3\u30b9\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u306e PyTorch \u5b9f\u9a13\u3067\u3059\u3002</p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52d5\u56de\u5e30\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configurations can and will be overridden when we start the experiment.</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u69cb\u6210\u306f\u3001\u5b9f\u9a13\u3092\u958b\u59cb\u3059\u308b\u3068\u304d\u306b\u4e0a\u66f8\u304d\u3067\u304d\u307e\u3059\u3002\u307e\u305f\u3001\u4eca\u5f8c\u5909\u66f4\u3059\u308b\u4e88\u5b9a\u3067\u3059\u3002</p>\n",
|
||||
"<h3>Initialize the attention reconstruction loss</h3>\n": "<h3>\u6ce8\u610f\u529b\u518d\u69cb\u7bc9\u30ed\u30b9\u3092\u521d\u671f\u5316</h3>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u3092\u521d\u671f\u5316</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u5b9f\u9a13\u3092\u5b9f\u884c\u3059\u308b</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u4e2d\u306b\u5b9a\u671f\u7684\u306b\u30b5\u30f3\u30d7\u30eb\u3092\u751f\u6210\u3059\u308b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u6a5f\u80fd</h3>\n",
|
||||
"<h3>Training/validation step</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0/\u691c\u8a3c\u30b9\u30c6\u30c3\u30d7</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Concatenate new memories and compress the oldest memories.</p>\n": "<p>\u65b0\u3057\u3044\u8a18\u61b6\u3092\u9023\u7d50\u3057\u3001\u6700\u3082\u53e4\u3044\u8a18\u61b6\u3092\u5727\u7e2e\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u3059\u308b\u8a2d\u5b9a\u306e\u8f9e\u66f8</p>\n",
|
||||
"<p>A list to keep memories that need to be compressed for each layer. </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u3054\u3068\u306b\u5727\u7e2e\u3059\u308b\u5fc5\u8981\u304c\u3042\u308b\u30e1\u30e2\u30ea\u3092\u4fdd\u5b58\u3059\u308b\u305f\u3081\u306e\u30ea\u30b9\u30c8\u3002</p>\n",
|
||||
"<p>A list to keep the memories that do not get compressed for each layer. </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u3054\u3068\u306b\u5727\u7e2e\u3055\u308c\u306a\u3044\u30e1\u30e2\u30ea\u3092\u4fdd\u5b58\u3059\u308b\u305f\u3081\u306e\u30ea\u30b9\u30c8\u3002</p>\n",
|
||||
"<p>Add a hook to log module outputs </p>\n": "<p>\u30e2\u30b8\u30e5\u30fc\u30eb\u51fa\u529b\u3092\u30ed\u30b0\u306b\u8a18\u9332\u3059\u308b\u30d5\u30c3\u30af\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Add attention reconstruction loss to loss </p>\n": "<p>\u640d\u5931\u306b\u6ce8\u610f\u518d\u69cb\u7bc9\u640d\u5931\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u30ed\u30ae\u30f3\u30b0\u7528\u306e\u4e88\u6e2c\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u4e88\u6e2c\u3092\u30d7\u30ed\u30f3\u30d7\u30c8\u306b\u8ffd\u52a0</p>\n",
|
||||
"<p>Attention Reconstruction Loss </p>\n": "<p>\u6ce8\u610f\u529b\u518d\u5efa\u30ed\u30b9</p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u7cbe\u5ea6\u306e\u8a08\u7b97\u3068\u8a18\u9332</p>\n",
|
||||
"<p>Calculate and log cross entropy loss </p>\n": "<p>\u30af\u30ed\u30b9\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u640d\u5931\u306e\u8a08\u7b97\u3068\u8a18\u9332</p>\n",
|
||||
"<p>Calculate attention reconstruction loss if memories were compressed in this step </p>\n": "<p>\u3053\u306e\u30b9\u30c6\u30c3\u30d7\u3067\u8a18\u61b6\u304c\u5727\u7e2e\u3055\u308c\u305f\u5834\u5408\u306e\u6ce8\u610f\u518d\u69cb\u6210\u640d\u5931\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u52fe\u914d\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate the number of compressed memories to make <span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is the number of memories we have and <span translate=no>_^_2_^_</span> is the maximum number of memories we maintain (<span translate=no>_^_3_^_</span>). </p>\n": "<p>\u4f5c\u6210\u3059\u308b\u5727\u7e2e\u30e1\u30e2\u30ea\u306e\u6570\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002\u3053\u3053\u3067<span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span>\u306f\u4fdd\u6301\u3059\u308b\u30e1\u30e2\u30ea\u306e\u6700\u5927\u6570\u3001<span translate=no>_^_2_^_</span>\u306f\u4fdd\u6301\u3059\u308b\u30e1\u30e2\u30ea\u306e\u6700\u5927\u6570 (<span translate=no>_^_3_^_</span>)\u3002</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30af\u30ea\u30a2</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u30af\u30ea\u30c3\u30d7\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Collect memories to compress </p>\n": "<p>\u601d\u3044\u51fa\u3092\u96c6\u3081\u3066\u5727\u7e2e</p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u5370\u5237\u7528\u306e\u51fa\u529b\u3092\u53ce\u96c6</p>\n",
|
||||
"<p>Collect remaining memories </p>\n": "<p>\u6b8b\u308a\u306e\u601d\u3044\u51fa\u3092\u96c6\u3081\u3088\u3046</p>\n",
|
||||
"<p>Compress the memories </p>\n": "<p>\u601d\u3044\u51fa\u3092\u5727\u7e2e</p>\n",
|
||||
"<p>Compress the oldest memories if there are more memories than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3088\u308a\u591a\u304f\u306e\u30e1\u30e2\u30ea\u304c\u3042\u308b\u5834\u5408\u306f\u3001\u6700\u3082\u53e4\u3044\u30e1\u30e2\u30ea\u3092\u5727\u7e2e\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Compressed memory length </p>\n": "<p>\u5727\u7e2e\u30e1\u30e2\u30ea\u9577</p>\n",
|
||||
"<p>Compression rate </p>\n": "<p>\u5727\u7e2e\u7387</p>\n",
|
||||
"<p>Concatenate new memories with old memory </p>\n": "<p>\u65b0\u3057\u3044\u8a18\u61b6\u3068\u53e4\u3044\u8a18\u61b6\u3092\u3064\u306a\u3052\u308b</p>\n",
|
||||
"<p>Concatenate newly compressed memories with old compressed memories </p>\n": "<p>\u65b0\u3057\u304f\u5727\u7e2e\u3055\u308c\u305f\u30e1\u30e2\u30ea\u3092\u53e4\u3044\u5727\u7e2e\u30e1\u30e2\u30ea\u3068\u9023\u7d50\u3059\u308b</p>\n",
|
||||
"<p>Concatenate the masks if there is memory </p>\n": "<p>\u30e1\u30e2\u30ea\u304c\u3042\u308b\u5834\u5408\u306f\u30de\u30b9\u30af\u3092\u9023\u7d50\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Create a subsequent mask for tokens </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u30de\u30b9\u30af\u3092\u5f8c\u304b\u3089\u4f5c\u6210</p>\n",
|
||||
"<p>Create an all ones (full visibility) mask for memory </p>\n": "<p>\u30e1\u30e2\u30ea\u7528\u306e\u30aa\u30fc\u30eb\u30ef\u30f3 (\u30d5\u30eb\u30d3\u30b8\u30d3\u30ea\u30c6\u30a3) \u30de\u30b9\u30af\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Do not print the attention reconstruction loss in the terminal </p>\n": "<p>\u7aef\u672b\u306b\u6ce8\u610f\u518d\u69cb\u6210\u30ed\u30b9\u3092\u5370\u5237\u3057\u306a\u3044\u3067\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8131\u843d\u78ba\u7387</p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u6700\u7d42\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u6b21\u306e\u30c8\u30fc\u30af\u30f3\u306e\u30ed\u30b8\u30c3\u30c8\u3092\u751f\u6210</p>\n",
|
||||
"<p>Get attention reconstruction loss </p>\n": "<p>\u6ce8\u610f\u3092\u5411\u3051\u3066\u518d\u5efa\u30ed\u30b9</p>\n",
|
||||
"<p>Get memories </p>\n": "<p>\u601d\u3044\u51fa\u3092\u30b2\u30c3\u30c8</p>\n",
|
||||
"<p>Get memory and compressed memory </p>\n": "<p>\u30e1\u30e2\u30ea\u3068\u5727\u7e2e\u30e1\u30e2\u30ea\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u30e2\u30c7\u30eb\u4e88\u6e2c\u3092\u53d6\u5f97 (\u6b32\u5f35\u308a)</p>\n",
|
||||
"<p>If the configurations specify not to use memory </p>\n": "<p>\u69cb\u6210\u3067\u30e1\u30e2\u30ea\u3092\u4f7f\u7528\u3057\u306a\u3044\u3088\u3046\u6307\u5b9a\u3055\u308c\u3066\u3044\u308b\u5834\u5408</p>\n",
|
||||
"<p>If there are no old compressed memories </p>\n": "<p>\u53e4\u3044\u5727\u7e2e\u30e1\u30e2\u30ea\u304c\u306a\u3044\u5834\u5408</p>\n",
|
||||
"<p>Iterate through memories of each layer. </p>\n": "<p>\u5404\u30ec\u30a4\u30e4\u30fc\u306e\u30e1\u30e2\u30ea\u3092\u7e70\u308a\u8fd4\u3057\u51e6\u7406\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u5404\u30a8\u30dd\u30c3\u30af\u306e\u6700\u5f8c\u306e\u30d0\u30c3\u30c1\u3067\u30e2\u30c7\u30eb\u30d1\u30e9\u30e1\u30fc\u30bf\u3068\u52fe\u914d\u3092\u8a18\u9332\u3057\u307e\u3059</p>\n",
|
||||
"<p>Masks </p>\n": "<p>\u30de\u30b9\u30af</p>\n",
|
||||
"<p>Merge and compress memory </p>\n": "<p>\u30e1\u30e2\u30ea\u306e\u7d71\u5408\u3068\u5727\u7e2e</p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u30c7\u30fc\u30bf\u3092\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Move to device </p>\n": "<p>\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>No memories are compressed if the number of memories is less than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30e1\u30e2\u30ea\u306e\u6570\u304c\u4ee5\u4e0b\u306e\u5834\u5408\u3001\u30e1\u30e2\u30ea\u306f\u5727\u7e2e\u3055\u308c\u307e\u305b\u3093 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u6570</p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN \u96a0\u308c\u30ec\u30a4\u30e4\u30fc\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u6570</p>\n",
|
||||
"<p>Number of memories to compress <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5727\u7e2e\u3059\u308b\u30e1\u30e2\u30ea\u306e\u6570 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of memories to keep </p>\n": "<p>\u4fdd\u5b58\u3059\u308b\u30e1\u30e2\u30ea\u306e\u6570</p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u5909\u5727\u5668\u5c64\u306e\u6570</p>\n",
|
||||
"<p>Only feed the last character to model in next iteration, rest will go in as memories </p>\n": "<p>\u6b21\u306e\u30a4\u30c6\u30ec\u30fc\u30b7\u30e7\u30f3\u3067\u306f\u6700\u5f8c\u306e\u6587\u5b57\u3060\u3051\u3092\u30e2\u30c7\u30eb\u306b\u30d5\u30a3\u30fc\u30c9\u3057\u3001\u6b8b\u308a\u306f\u30e1\u30e2\u30ea\u3068\u3057\u3066\u6b8b\u308a\u307e\u3059</p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u51fa\u529b\u3092\u5370\u5237\u3059\u308b</p>\n",
|
||||
"<p>Return memories and the memories that were compressed. Memories that were compressed are needed for the reconstruction loss computation. </p>\n": "<p>\u30e1\u30e2\u30ea\u3068\u5727\u7e2e\u3055\u308c\u305f\u30e1\u30e2\u30ea\u3092\u8fd4\u3057\u307e\u3059\u3002\u518d\u69cb\u6210\u640d\u5931\u306e\u8a08\u7b97\u306b\u306f\u3001\u5727\u7e2e\u3055\u308c\u305f\u30e1\u30e2\u30ea\u304c\u5fc5\u8981\u3067\u3059</p>\u3002\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u5909\u5727\u5668\u306b\u901a\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Run the model </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>25\u30c8\u30fc\u30af\u30f3\u306e\u30b5\u30f3\u30d7\u30eb</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u8ffd\u8de1\u3057\u305f\u30e1\u30c8\u30ea\u30af\u30b9\u3092\u4fdd\u5b58\u3059\u308b</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u30c8\u30e9\u30c3\u30ab\u30fc\u69cb\u6210\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Split the memories at <span translate=no>_^_0_^_</span> </p>\n": "<p>\u601d\u3044\u51fa\u3092\u5206\u3051\u3066 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u8d77\u52d5\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>State module to maintain memories when switching between training and validation </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b\u3068\u304d\u306b\u30e1\u30e2\u30ea\u3092\u7dad\u6301\u3059\u308b\u30b9\u30c6\u30fc\u30c8\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u6700\u9069\u5316\u306e\u4e00\u6b69\u3092\u8e0f\u307f\u51fa\u3059</p>\n",
|
||||
"<p>This will keep the accuracy metric stats and memories separate for training and validation. </p>\n": "<p>\u3053\u308c\u306b\u3088\u308a\u3001\u7cbe\u5ea6\u30e1\u30c8\u30ea\u30c3\u30af\u306e\u7d71\u8a08\u60c5\u5831\u3068\u30e1\u30e2\u30ea\u304c\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u7528\u306b\u5225\u3005\u306b\u4fdd\u6301\u3055\u308c\u307e\u3059\u3002</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30e2\u30b8\u30e5\u30fc\u30eb</p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f</p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u3092\u30c8\u30fc\u30af\u30f3\u5316</p>\n",
|
||||
"<p>Total length of the memory and compressed memory (for masks) </p>\n": "<p>\u30e1\u30e2\u30ea\u3068\u5727\u7e2e\u30e1\u30e2\u30ea\u306e\u5408\u8a08\u9577 (\u30de\u30b9\u30af\u7528)</p>\n",
|
||||
"<p>Track attention reconstruction loss </p>\n": "<p>\u30c8\u30e9\u30c3\u30af\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30ea\u30b3\u30f3\u30b9\u30c8\u30e9\u30af\u30b7\u30e7\u30f3\u30fb\u30ed\u30b9</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u5909\u5727\u5668</p>\n",
|
||||
"<p>Truncate old memories </p>\n": "<p>\u53e4\u3044\u601d\u3044\u51fa\u3092\u5207\u308a\u6368\u3066\u308b</p>\n",
|
||||
"<p>Update and compress memory </p>\n": "<p>\u30e1\u30e2\u30ea\u306e\u66f4\u65b0\u3068\u5727\u7e2e</p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30e2\u30fc\u30c9\u6642\u306b\u30b0\u30ed\u30fc\u30d0\u30eb\u30b9\u30c6\u30c3\u30d7 (\u51e6\u7406\u3055\u308c\u305f\u30c8\u30fc\u30af\u30f3\u306e\u6570) \u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Update memories </p>\n": "<p>\u30e1\u30e2\u30ea\u30fc\u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Update the memories </p>\n": "<p>\u601d\u3044\u51fa\u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Use only the subsequent mask otherwise </p>\n": "<p>\u305d\u308c\u4ee5\u5916\u306e\u5834\u5408\u306f\u3001\u5f8c\u7d9a\u306e\u30de\u30b9\u30af\u306e\u307f\u3092\u4f7f\u7528\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u30ad\u30e3\u30d7\u30c1\u30e3\u3059\u308b\u304b\u3069\u3046\u304b</p>\n",
|
||||
"<p>memory </p>\n": "<p>\u8a18\u61b6</p>\n",
|
||||
"Compressive Transformer Experiment": "\u5727\u7e2e\u5909\u5727\u5668\u5b9f\u9a13",
|
||||
"This experiment trains a compressive transformer model on tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u5c0f\u3055\u306a\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067\u5727\u7e2e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"<h1>Compressive Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a compressive transformer model.</p>\n": "<h1>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configurations can and will be overridden when we start the experiment.</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0d85\u0db4 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1 \u0dc0\u0dd2\u0da7 \u0db4\u0dd9\u0dbb\u0db1\u0dd2\u0db8\u0dd2 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0d89\u0d9a\u0dca\u0db8\u0dc0\u0dcf \u0dba\u0dcf \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. </p>\n",
|
||||
"<h3>Initialize the attention reconstruction loss</h3>\n": "<h3>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0dc0\u0dbb\u0dd2\u0db1\u0dca \u0dc0\u0dbb \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba</h3>\n",
|
||||
"<h3>Training/validation step</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0/\u0dc0\u0dbd\u0d82\u0d9c\u0dd4\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db4\u0dd2\u0dba\u0dc0\u0dbb</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Concatenate new memories and compress the oldest memories.</p>\n": "<p> \u0db1\u0dc0\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb \u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dad\u0db8 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf\u0dba\u0dcf\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 \u0dc1\u0db6\u0dca\u0daf\u0d9a\u0ddd\u0dc2\u0dba\u0d9a\u0dca </p>\n",
|
||||
"<p>A list to keep memories that need to be compressed for each layer. </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dad\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0dca. </p>\n",
|
||||
"<p>A list to keep the memories that do not get compressed for each layer. </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db1\u0ddc\u0dc0\u0db1 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dad\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0dca. </p>\n",
|
||||
"<p>Add a hook to log module outputs </p>\n": "<p>\u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d9a\u0ddc\u0d9a\u0dca\u0d9a\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add attention reconstruction loss to loss </p>\n": "<p>\u0d85\u0dbd\u0dcf\u0db7\u0dba\u0da7\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u0dbd\u0ddc\u0d9c\u0dca\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0da7\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Attention Reconstruction Loss </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab \u0d85\u0dbd\u0dcf\u0db7\u0dba </p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf\u0dc0\u0dba </p>\n",
|
||||
"<p>Calculate and log cross entropy loss </p>\n": "<p>\u0dc4\u0dbb\u0dc3\u0dca\u0d91\u0db1\u0dca\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd2 \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate attention reconstruction loss if memories were compressed in this step </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0db8\u0dd9\u0db8 \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db1\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0d9c\u0dab\u0db1\u0dba </p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate the number of compressed memories to make <span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is the number of memories we have and <span translate=no>_^_2_^_</span> is the maximum number of memories we maintain (<span translate=no>_^_3_^_</span>). </p>\n": "<p>\u0dc3\u0dd1\u0daf\u0dd3\u0db8\u0da7\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span>, \u0d85\u0db4 \u0dc3\u0dad\u0dd4\u0dc0 \u0d87\u0dad\u0dd2 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 <span translate=no>_^_1_^_</span> \u0d9a\u0ddc\u0dad\u0dd0\u0db1\u0daf \u0dc3\u0dc4 <span translate=no>_^_2_^_</span> \u0d85\u0db4 \u0db1\u0da9\u0dad\u0dca\u0dad\u0dd4 \u0d9a\u0dbb\u0db1 \u0d8b\u0db4\u0dbb\u0dd2\u0db8 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 ( <span translate=no>_^_3_^_</span>). </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d89\u0dc0\u0dad\u0dca </p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dca\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a </p>\n",
|
||||
"<p>Collect memories to compress </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Collect remaining memories </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Compress the memories </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dc3\u0d82\u0d9a\u0ddd\u0da0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Compress the oldest memories if there are more memories than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc0\u0da9\u0dcf\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dad\u0dd2\u0db6\u0dda \u0db1\u0db8\u0dca \u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dad\u0db8 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Compressed memory length </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad\u0db8\u0dad\u0d9a \u0daf\u0dd2\u0d9c </p>\n",
|
||||
"<p>Compression rate </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba </p>\n",
|
||||
"<p>Concatenate new memories with old memory </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0db8\u0dad\u0d9a\u0dba \u0dc3\u0db8\u0d9f \u0db1\u0dc0 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Concatenate newly compressed memories with old compressed memories </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0d9f \u0d85\u0dbd\u0dd4\u0dad\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Concatenate the masks if there is memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0d9a\u0dca\u0dad\u0dd2\u0db6\u0dda \u0db1\u0db8\u0dca \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create a subsequent mask for tokens </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create an all ones (full visibility) mask for memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0dd2\u0dba\u0dbd\u0dd4 (\u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab \u0daf\u0dd8\u0dc1\u0dca\u0dba\u0dad\u0dcf\u0dc0) \u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Do not print the attention reconstruction loss in the terminal </p>\n": "<p>\u0db4\u0dbb\u0dca\u0dba\u0db1\u0dca\u0dad\u0dba\u0dda\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0db1\u0ddc\u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u0d85\u0dad\u0dc4\u0dd0\u0dbb\u0daf\u0dd0\u0db8\u0dd3\u0db8\u0dda \u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 </p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0db1\u0dca\u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get attention reconstruction loss </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get memories </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get memory and compressed memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0dc3\u0dc4 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 (\u0d9a\u0dd1\u0daf\u0dbb) </p>\n",
|
||||
"<p>If the configurations specify not to use memory </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db8\u0dad\u0d9a\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0db1\u0ddc\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db1\u0dd2\u0dba\u0db8 \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dda \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>If there are no old compressed memories </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>Iterate through memories of each layer. </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc4\u0dbb\u0dc4\u0dcf \u0d9c\u0db8\u0db1\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u0dc3\u0dd1\u0db8\u0dba\u0dd4\u0d9c\u0dbd\u0dba\u0d9a\u0db8 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dda \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0dc4 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Masks </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 </p>\n",
|
||||
"<p>Merge and compress memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0d92\u0d9a\u0dcf\u0db6\u0daf\u0dca\u0db0 \u0d9a\u0dbb \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dc0\u0dd9\u0dad \u0daf\u0dad\u0dca\u0dad \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move to device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dc0\u0dd9\u0dad \u0d9c\u0dd9\u0db1 \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>No memories are compressed if the number of memories is less than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0d9c\u0dab\u0db1 \u0d85\u0da9\u0dd4 \u0db1\u0db8\u0dca \u0d9a\u0dd2\u0dc3\u0dd2\u0daf\u0dd4 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db1\u0ddc\u0dc0\u0dda <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN\u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of memories to compress <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Number of memories to keep </p>\n": "<p>\u0dad\u0db6\u0dcf\u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc3\u0dca\u0dae\u0dbb \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Only feed the last character to model in next iteration, rest will go in as memories </p>\n": "<p>\u0d8a\u0dc5\u0d9f\u0db4\u0dd4\u0db1\u0dbb\u0dcf\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0dda\u0daf\u0dd3 \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0da0\u0dbb\u0dd2\u0dad\u0dba \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0da7 \u0db4\u0db8\u0dab\u0d9a\u0dca \u0db4\u0ddd\u0dc2\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1, \u0dc0\u0dd2\u0dc0\u0dda\u0d9a\u0dba \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dbd\u0dd9\u0dc3 \u0d89\u0daf\u0dd2\u0dbb\u0dd2\u0dba\u0da7 \u0dba\u0db1\u0dd4 \u0d87\u0dad </p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Return memories and the memories that were compressed. Memories that were compressed are needed for the reconstruction loss computation. </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dc3\u0dc4 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1\u0dca\u0db1. \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db6\u0dc0 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab\u0dba \u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dda. </p>\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dba\u0dc4\u0dbb\u0dc4\u0dcf \u0d91\u0dba \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd25 \u0da7\u0ddd\u0d9a\u0db1 </p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u0dbd\u0dd4\u0dc4\u0dd4\u0db6\u0dd0\u0db3\u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dd2\u0dad\u0dd2\u0d9a \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dd0\u0d9a\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Split the memories at <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0daf\u0dd3\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db6\u0dd9\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>State module to maintain memories when switching between training and validation </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0\u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4\u0dc0\u0dd3\u0db8\u0dda\u0daf\u0dd3 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0db4\u0dc0\u0dad\u0dca\u0dc0\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>This will keep the accuracy metric stats and memories separate for training and validation. </p>\n": "<p>\u0db8\u0dd9\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dbb\u0dc0\u0daf\u0dca\u0dba\u0dad\u0dcf \u0db8\u0dd9\u0da7\u0dca\u0dbb\u0dd2\u0d9a\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0db1 \u0dc3\u0dc4 \u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0dc0\u0dd9\u0db1\u0db8 \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dd3. </p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba </p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dda \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca </p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0da7\u0ddd\u0d9a\u0dd9\u0db1\u0dca\u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Total length of the memory and compressed memory (for masks) </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0dda\u0db8\u0dd4\u0dc5\u0dd4 \u0daf\u0dd2\u0d9c \u0dc3\u0dc4 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dd2\u0dad \u0db8\u0dad\u0d9a\u0dba\u0dda (\u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dc3\u0db3\u0dc4\u0dcf) </p>\n",
|
||||
"<p>Track attention reconstruction loss </p>\n": "<p>\u0db0\u0dcf\u0dc0\u0db1\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dc3\u0d82\u0dc3\u0dca\u0d9a\u0dbb\u0dab \u0d85\u0dbd\u0dcf\u0db7\u0dba </p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca </p>\n",
|
||||
"<p>Truncate old memories </p>\n": "<p>\u0db4\u0dd0\u0dbb\u0dab\u0dd2\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Update and compress memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db4\u0dca\u0dbb\u0d9a\u0dcf\u0dbb\u0dba\u0dda\u0daf\u0dd3 \u0d9c\u0ddd\u0dbd\u0dd3\u0dba \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 (\u0dc3\u0dd0\u0d9a\u0dc3\u0dd6 \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1) </p>\n",
|
||||
"<p>Update memories </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Update the memories </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba\u0db1\u0dca\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use only the subsequent mask otherwise </p>\n": "<p>\u0dc0\u0dd9\u0db1\u0dad\u0dca\u0d86\u0d9a\u0dcf\u0dbb\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0db4\u0dc3\u0dd4\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d86\u0dc0\u0dbb\u0dab \u0db4\u0db8\u0dab\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0d9c\u0dca\u0dbb\u0dc4\u0dab\u0dba \u0d9a\u0dbb \u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4\u0daf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>memory </p>\n": "<p>\u0db8\u0dad\u0d9a\u0dba </p>\n",
|
||||
"Compressive Transformer Experiment": "\u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains a compressive transformer model on tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db8\u0dad \u0dc3\u0db8\u0dca\u0db4\u0dd3\u0da9\u0dca\u0dba\u0dad\u0dcf \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"<h1>Compressive Transformer Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a compressive transformer model.</p>\n": "<h1>\u538b\u7f29\u53d8\u538b\u5668\u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u4e2a\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3\u538b\u7f29\u53d8\u538b\u5668\u6a21\u578b\u3002</p>\n",
|
||||
"<h2>Auto regressive model</h2>\n": "<h2>\u81ea\u52a8\u56de\u5f52\u6a21\u578b</h2>\n",
|
||||
"<h2>Configurations</h2>\n<p>The default configurations can and will be overridden when we start the experiment.</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u5f53\u6211\u4eec\u5f00\u59cb\u5b9e\u9a8c\u65f6\uff0c\u9ed8\u8ba4\u914d\u7f6e\u53ef\u4ee5\u800c\u4e14\u5c06\u4f1a\u88ab\u8986\u76d6\u3002</p>\n",
|
||||
"<h3>Initialize the attention reconstruction loss</h3>\n": "<h3>\u521d\u59cb\u5316\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</h3>\n",
|
||||
"<h3>Initialize the auto-regressive model</h3>\n": "<h3>\u521d\u59cb\u5316\u81ea\u56de\u5f52\u6a21\u578b</h3>\n",
|
||||
"<h3>Run the experiment</h3>\n": "<h3>\u8fd0\u884c\u5b9e\u9a8c</h3>\n",
|
||||
"<h3>Sampling function to generate samples periodically while training</h3>\n": "<h3>\u91c7\u6837\u529f\u80fd\u53ef\u5728\u8bad\u7ec3\u65f6\u5b9a\u671f\u751f\u6210\u6837\u672c</h3>\n",
|
||||
"<h3>Training/validation step</h3>\n": "<h3>\u57f9\u8bad/\u9a8c\u8bc1\u6b65\u9aa4</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Concatenate new memories and compress the oldest memories.</p>\n": "<p>\u8fde\u63a5\u65b0\u8bb0\u5fc6\u5e76\u538b\u7f29\u6700\u53e4\u8001\u7684\u8bb0\u5fc6\u3002</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A dictionary of configurations to override </p>\n": "<p>\u8981\u8986\u76d6\u7684\u914d\u7f6e\u5b57\u5178</p>\n",
|
||||
"<p>A list to keep memories that need to be compressed for each layer. </p>\n": "<p>\u7528\u4e8e\u4fdd\u5b58\u6bcf\u5c42\u9700\u8981\u538b\u7f29\u7684\u5185\u5b58\u7684\u5217\u8868\u3002</p>\n",
|
||||
"<p>A list to keep the memories that do not get compressed for each layer. </p>\n": "<p>\u4e00\u4e2a\u5217\u8868\uff0c\u7528\u4e8e\u4fdd\u5b58\u6bcf\u5c42\u672a\u88ab\u538b\u7f29\u7684\u8bb0\u5fc6\u3002</p>\n",
|
||||
"<p>Add a hook to log module outputs </p>\n": "<p>\u5411\u65e5\u5fd7\u6a21\u5757\u8f93\u51fa\u6dfb\u52a0\u94a9\u5b50</p>\n",
|
||||
"<p>Add attention reconstruction loss to loss </p>\n": "<p>\u5c06\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931\u589e\u52a0\u5230\u635f\u5931</p>\n",
|
||||
"<p>Add the prediction for logging </p>\n": "<p>\u6dfb\u52a0\u65e5\u5fd7\u8bb0\u5f55\u7684\u9884\u6d4b</p>\n",
|
||||
"<p>Add the prediction to prompt </p>\n": "<p>\u5c06\u9884\u6d4b\u6dfb\u52a0\u5230\u63d0\u793a\u7b26\u4e2d</p>\n",
|
||||
"<p>Attention Reconstruction Loss </p>\n": "<p>\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Calculate and log accuracy </p>\n": "<p>\u8ba1\u7b97\u548c\u8bb0\u5f55\u7cbe\u5ea6</p>\n",
|
||||
"<p>Calculate and log cross entropy loss </p>\n": "<p>\u8ba1\u7b97\u548c\u8bb0\u5f55\u4ea4\u53c9\u71b5\u635f\u5931</p>\n",
|
||||
"<p>Calculate attention reconstruction loss if memories were compressed in this step </p>\n": "<p>\u5982\u679c\u5728\u6b64\u6b65\u9aa4\u4e2d\u8bb0\u5fc6\u88ab\u538b\u7f29\uff0c\u5219\u8ba1\u7b97\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"<p>Calculate the number of compressed memories to make <span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is the number of memories we have and <span translate=no>_^_2_^_</span> is the maximum number of memories we maintain (<span translate=no>_^_3_^_</span>). </p>\n": "<p>\u8ba1\u7b97\u8981\u5236\u4f5c\u7684\u538b\u7f29\u8bb0\u5fc6\u7684\u6570\u91cf<span translate=no>_^_0_^_</span>\uff0c\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u6211\u4eec\u62e5\u6709\u7684\u8bb0\u5fc6\u6570\u91cf\uff0c<span translate=no>_^_2_^_</span>\u662f\u6211\u4eec\u7ef4\u62a4\u7684\u6700\u5927\u8bb0\u5fc6\u6570\uff08<span translate=no>_^_3_^_</span>)\u3002</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u6e05\u9664\u6e10\u53d8</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u526a\u8f91\u6e10\u53d8</p>\n",
|
||||
"<p>Collect memories to compress </p>\n": "<p>\u6536\u96c6\u8bb0\u5fc6\u8fdb\u884c\u538b\u7f29</p>\n",
|
||||
"<p>Collect output for printing </p>\n": "<p>\u6536\u96c6\u8f93\u51fa\u4ee5\u8fdb\u884c\u6253\u5370</p>\n",
|
||||
"<p>Collect remaining memories </p>\n": "<p>\u6536\u96c6\u5269\u4f59\u7684\u8bb0\u5fc6</p>\n",
|
||||
"<p>Compress the memories </p>\n": "<p>\u538b\u7f29\u8bb0\u5fc6</p>\n",
|
||||
"<p>Compress the oldest memories if there are more memories than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u8bb0\u5fc6\u591a\u4e8e\u6700\u65e9\u7684\u8bb0\u5fc6\uff0c\u5219\u538b\u7f29\u6700\u65e9\u7684\u8bb0\u5fc6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Compressed memory length </p>\n": "<p>\u538b\u7f29\u7684\u5185\u5b58\u957f\u5ea6</p>\n",
|
||||
"<p>Compression rate </p>\n": "<p>\u538b\u7f29\u7387</p>\n",
|
||||
"<p>Concatenate new memories with old memory </p>\n": "<p>\u5c06\u65b0\u8bb0\u5fc6\u4e0e\u65e7\u8bb0\u5fc6\u8fde\u63a5\u8d77\u6765</p>\n",
|
||||
"<p>Concatenate newly compressed memories with old compressed memories </p>\n": "<p>\u5c06\u65b0\u538b\u7f29\u7684\u5b58\u50a8\u5668\u4e0e\u65e7\u7684\u538b\u7f29\u5b58\u50a8\u5668\u8fde\u63a5\u8d77\u6765</p>\n",
|
||||
"<p>Concatenate the masks if there is memory </p>\n": "<p>\u5982\u679c\u6709\u5185\u5b58\uff0c\u5219\u8fde\u63a5\u63a9\u7801</p>\n",
|
||||
"<p>Create a subsequent mask for tokens </p>\n": "<p>\u4e3a\u4ee4\u724c\u521b\u5efa\u540e\u7eed\u63a9\u7801</p>\n",
|
||||
"<p>Create an all ones (full visibility) mask for memory </p>\n": "<p>\u4e3a\u5185\u5b58\u521b\u5efa\u4e00\u4e2a\u5168\u4e00\uff08\u5b8c\u5168\u53ef\u89c1\u6027\uff09\u63a9\u7801</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Do not print the attention reconstruction loss in the terminal </p>\n": "<p>\u4e0d\u8981\u5728\u7ec8\u7aef\u4e2d\u6253\u5370\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Dropout probability </p>\n": "<p>\u8f8d\u5b66\u6982\u7387</p>\n",
|
||||
"<p>Final layer </p>\n": "<p>\u6700\u540e\u4e00\u5c42</p>\n",
|
||||
"<p>Generate logits of the next token </p>\n": "<p>\u751f\u6210\u4e0b\u4e00\u4e2a\u4ee4\u724c\u7684\u65e5\u5fd7</p>\n",
|
||||
"<p>Get attention reconstruction loss </p>\n": "<p>\u5f15\u8d77\u6ce8\u610f\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Get memories </p>\n": "<p>\u83b7\u5f97\u56de\u5fc6</p>\n",
|
||||
"<p>Get memory and compressed memory </p>\n": "<p>\u83b7\u53d6\u5185\u5b58\u548c\u538b\u7f29\u5185\u5b58</p>\n",
|
||||
"<p>Get the model output </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>Get the model prediction (greedy) </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u9884\u6d4b\uff08\u8d2a\u5a6a\uff09</p>\n",
|
||||
"<p>If the configurations specify not to use memory </p>\n": "<p>\u5982\u679c\u914d\u7f6e\u6307\u5b9a\u4e0d\u4f7f\u7528\u5185\u5b58</p>\n",
|
||||
"<p>If there are no old compressed memories </p>\n": "<p>\u5982\u679c\u6ca1\u6709\u65e7\u7684\u538b\u7f29\u8bb0\u5fc6</p>\n",
|
||||
"<p>Iterate through memories of each layer. </p>\n": "<p>\u904d\u5386\u6bcf\u5c42\u7684\u8bb0\u5fc6\u3002</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Log the model parameters and gradients on last batch of every epoch </p>\n": "<p>\u8bb0\u5f55\u6bcf\u4e2a\u7eaa\u5143\u6700\u540e\u4e00\u6279\u7684\u6a21\u578b\u53c2\u6570\u548c\u68af\u5ea6</p>\n",
|
||||
"<p>Masks </p>\n": "<p>\u53e3\u7f69</p>\n",
|
||||
"<p>Merge and compress memory </p>\n": "<p>\u5408\u5e76\u548c\u538b\u7f29\u5185\u5b58</p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u5c06\u6570\u636e\u79fb\u52a8\u5230\u8bbe\u5907</p>\n",
|
||||
"<p>Move to device </p>\n": "<p>\u79fb\u81f3\u8bbe\u5907</p>\n",
|
||||
"<p>No memories are compressed if the number of memories is less than <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u5185\u5b58\u6570\u91cf\u5c11\u4e8e<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of attention heads </p>\n": "<p>\u6ce8\u610f\u5934\u6570\u91cf</p>\n",
|
||||
"<p>Number of features in FFN hidden layer </p>\n": "<p>FFN \u9690\u85cf\u5c42\u4e2d\u7684\u8981\u7d20\u6570\u91cf</p>\n",
|
||||
"<p>Number of memories to compress <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8981\u538b\u7f29\u7684\u5185\u5b58\u6570\u91cf<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Number of memories to keep </p>\n": "<p>\u8981\u4fdd\u7559\u7684\u8bb0\u5fc6\u6570\u91cf</p>\n",
|
||||
"<p>Number of transformer layers </p>\n": "<p>\u53d8\u538b\u5668\u5c42\u6570</p>\n",
|
||||
"<p>Only feed the last character to model in next iteration, rest will go in as memories </p>\n": "<p>\u5728\u4e0b\u4e00\u6b21\u8fed\u4ee3\u4e2d\u53ea\u5582\u6700\u540e\u4e00\u4e2a\u89d2\u8272\u8fdb\u884c\u5efa\u6a21\uff0c\u5176\u4f59\u90e8\u5206\u5c06\u4f5c\u4e3a\u8bb0\u5fc6\u8fdb\u53bb</p>\n",
|
||||
"<p>Print the sampled output </p>\n": "<p>\u6253\u5370\u91c7\u6837\u8f93\u51fa</p>\n",
|
||||
"<p>Return memories and the memories that were compressed. Memories that were compressed are needed for the reconstruction loss computation. </p>\n": "<p>\u8fd4\u56de\u88ab\u538b\u7f29\u7684\u8bb0\u5fc6\u548c\u8bb0\u5fc6\u3002\u91cd\u5efa\u635f\u5931\u8ba1\u7b97\u9700\u8981\u88ab\u538b\u7f29\u7684\u8bb0\u5fc6\u3002</p>\n",
|
||||
"<p>Run it through the transformer </p>\n": "<p>\u7528\u5b83\u7a7f\u8fc7\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Run the model </p>\n": "<p>\u8fd0\u884c\u6a21\u578b</p>\n",
|
||||
"<p>Sample 25 tokens </p>\n": "<p>\u6837\u672c 25 \u4e2a\u4ee3\u5e01</p>\n",
|
||||
"<p>Save the tracked metrics </p>\n": "<p>\u4fdd\u5b58\u8ddf\u8e2a\u7684\u6307\u6807</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Set tracker configurations </p>\n": "<p>\u8bbe\u7f6e\u8ddf\u8e2a\u5668\u914d\u7f6e</p>\n",
|
||||
"<p>Split the memories at <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5728\u4ee5\u4e0b\u4f4d\u7f6e\u62c6\u5206\u8bb0\u5fc6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt </p>\n": "<p>\u542f\u52a8\u63d0\u793a</p>\n",
|
||||
"<p>State module to maintain memories when switching between training and validation </p>\n": "<p>\u72b6\u6001\u6a21\u5757\u7528\u4e8e\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u65f6\u4fdd\u6301\u8bb0\u5fc6</p>\n",
|
||||
"<p>Take optimizer step </p>\n": "<p>\u91c7\u53d6\u4f18\u5316\u5668\u6b65\u9aa4</p>\n",
|
||||
"<p>This will keep the accuracy metric stats and memories separate for training and validation. </p>\n": "<p>\u8fd9\u5c06\u4f7f\u7cbe\u5ea6\u6307\u6807\u7edf\u8ba1\u6570\u636e\u548c\u8bb0\u5fc6\u5206\u5f00\uff0c\u4ee5\u4fbf\u8bad\u7ec3\u548c\u9a8c\u8bc1\u3002</p>\n",
|
||||
"<p>Token embedding module </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u6a21\u5757</p>\n",
|
||||
"<p>Token embedding size </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u5927\u5c0f</p>\n",
|
||||
"<p>Token embeddings </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>Tokenize the prompt </p>\n": "<p>\u5c06\u63d0\u793a\u7b26\u53f7\u5316</p>\n",
|
||||
"<p>Total length of the memory and compressed memory (for masks) </p>\n": "<p>\u5185\u5b58\u548c\u538b\u7f29\u5185\u5b58\u7684\u603b\u957f\u5ea6\uff08\u7528\u4e8e\u63a9\u7801\uff09</p>\n",
|
||||
"<p>Track attention reconstruction loss </p>\n": "<p>\u8ffd\u8e2a\u6ce8\u610f\u529b\u91cd\u5efa\u635f\u5931</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u8bad\u7ec3\u6a21\u578b</p>\n",
|
||||
"<p>Transformer </p>\n": "<p>\u53d8\u538b\u5668</p>\n",
|
||||
"<p>Truncate old memories </p>\n": "<p>\u622a\u65ad\u65e7\u7684\u8bb0\u5fc6</p>\n",
|
||||
"<p>Update and compress memory </p>\n": "<p>\u66f4\u65b0\u548c\u538b\u7f29\u5185\u5b58</p>\n",
|
||||
"<p>Update global step (number of tokens processed) when in training mode </p>\n": "<p>\u5728\u8bad\u7ec3\u6a21\u5f0f\u4e0b\u66f4\u65b0\u5168\u5c40\u6b65\u957f\uff08\u5904\u7406\u7684\u4ee4\u724c\u6570\uff09</p>\n",
|
||||
"<p>Update memories </p>\n": "<p>\u66f4\u65b0\u8bb0\u5fc6</p>\n",
|
||||
"<p>Update the memories </p>\n": "<p>\u66f4\u65b0\u8bb0\u5fc6</p>\n",
|
||||
"<p>Use only the subsequent mask otherwise </p>\n": "<p>\u5426\u5219\uff0c\u4ec5\u4f7f\u7528\u540e\u7eed\u7684\u63a9\u7801</p>\n",
|
||||
"<p>Whether to capture model outputs </p>\n": "<p>\u662f\u5426\u6355\u83b7\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>memory </p>\n": "<p>\u8bb0\u5fc6</p>\n",
|
||||
"Compressive Transformer Experiment": "\u538b\u7f29\u53d8\u538b\u5668\u5b9e\u9a8c",
|
||||
"This experiment trains a compressive transformer model on tiny Shakespeare dataset.": "\u8fd9\u4e2a\u5b9e\u9a8c\u5728\u5fae\u5c0f\u7684\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u4e00\u4e2a\u538b\u7f29\u53d8\u538b\u5668\u6a21\u578b\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user