chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings (RoPE)</h1>\n<p>This is an implementation of <a href=\"https://arxiv.org/abs/2104.09864\">Rotary Positional Embeddings (RoPE)</a> in <a href=\"https://pytorch.org\">PyTorch</a>.</p>\n<p>Rotary Positional Embeddings (RoPE) encode position information of tokens with a rotation matrix that naturally incorporates explicit relative position dependency.</p>\n<p>Here's <a href=\"experiment.html\">the training code</a> for training a transformer model with RoPE on Tiny Shakespeare dataset.</p>\n": "<h1>\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (RoPE)</h1>\n<p>\u8fd9\u662f PyT <a href=\"https://pytorch.org\">orch \u4e2d\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (RoP</a> <a href=\"https://arxiv.org/abs/2104.09864\">E)</a> \u7684\u5b9e\u73b0\u3002</p>\n<p>Rotary Positional Embeddings (RoPE) \u4f7f\u7528\u81ea\u7136\u5305\u542b\u660e\u786e\u7684\u76f8\u5bf9\u4f4d\u7f6e\u4f9d\u8d56\u5173\u7cfb\u7684\u65cb\u8f6c\u77e9\u9635\u5bf9\u4ee3\u5e01\u7684\u4f4d\u7f6e\u4fe1\u606f\u8fdb\u884c\u7f16\u7801\u3002</p>\n<p>\u4ee5\u4e0b\u662f<a href=\"experiment.html\">\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u4f7f\u7528 RoPE \u8bad\u7ec3\u53d8\u538b\u5668\u6a21\u578b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
|
||||
"<h2>Multi-head attention with rotary positional embeddings</h2>\n<p>We override <a href=\"../mha.html\">multi-head attention from original transformer</a>.</p>\n": "<h2>\u901a\u8fc7\u65cb\u8f6c\u5b9a\u4f4d\u5d4c\u5165\u5b9e\u73b0\u591a\u5934\u5173\u6ce8</h2>\n<p>\u6211\u4eec\u8d85\u8d8a\u4e86<a href=\"../mha.html\">\u539f\u88c5\u53d8\u538b\u5668\u7684\u591a\u5934\u6ce8\u610f\u529b</a>\u3002</p>\n",
|
||||
"<h2>RoPE module</h2>\n<p>Rotary encoding transforms pairs of features by rotating in the 2D plane. That is, it organizes the <span translate=no>_^_0_^_</span> features as <span translate=no>_^_1_^_</span> pairs. Each pair can be considered a coordinate in a 2D plane, and the encoding will rotate it by an angle depending on the position of the token.</p>\n<h3>For a pair of features</h3>\n<p>Let <span translate=no>_^_2_^_</span> and <span translate=no>_^_3_^_</span> be two features of the key or query of any head at position <span translate=no>_^_4_^_</span>. Or for simplicity assume <span translate=no>_^_5_^_</span> has only two features. Then the transformation is,</p>\n<span translate=no>_^_6_^_</span><p>where <span translate=no>_^_7_^_</span> is a constant angle. The other pairs of features are transformed similarly.</p>\n<h3>Attention is relative</h3>\n<p>For a pair of features, dot-product attention score between two positions <span translate=no>_^_8_^_</span> and <span translate=no>_^_9_^_</span> would be</p>\n<span translate=no>_^_10_^_</span><p>This shows that for dot-production attention the rotary encodings gives relative attention.</p>\n<h3>For all features</h3>\n<p>The features are grouped into pairs and handled as above. They use a different <span translate=no>_^_11_^_</span> for each pair.</p>\n<p>The paper suggests using <span translate=no>_^_12_^_</span> for the <span translate=no>_^_13_^_</span> pairs of features.</p>\n<p>We pair feature <span translate=no>_^_14_^_</span> with feature <span translate=no>_^_15_^_</span>. So for position <span translate=no>_^_16_^_</span> we transform</p>\n<span translate=no>_^_17_^_</span><p>to</p>\n<span translate=no>_^_18_^_</span>": "<h2>\u7ef3\u7d22\u6a21\u5757</h2>\n<p>\u65cb\u8f6c\u7f16\u7801\u901a\u8fc7\u5728 2D \u5e73\u9762\u4e2d\u65cb\u8f6c\u6765\u8f6c\u6362\u6210\u5bf9\u7684\u8981\u7d20\u3002\u4e5f\u5c31\u662f\u8bf4\uff0c\u5b83\u5c06<span translate=no>_^_0_^_</span>\u8981\u7d20\u7ec4\u7ec7\u6210<span translate=no>_^_1_^_</span>\u5bf9\u3002\u6bcf\u5bf9\u90fd\u53ef\u4ee5\u88ab\u89c6\u4e3a\u4e8c\u7ef4\u5e73\u9762\u4e2d\u7684\u4e00\u4e2a\u5750\u6807\uff0c\u7f16\u7801\u5c06\u6839\u636e\u4ee4\u724c\u7684\u4f4d\u7f6e\u5c06\u5176\u65cb\u8f6c\u4e00\u4e2a\u89d2\u5ea6\u3002</p>\n<h3>\u5bf9\u4e8e\u4e00\u5bf9\u529f\u80fd</h3>\n<p>\u8ba9<span translate=no>_^_2_^_</span>\u548c<span translate=no>_^_3_^_</span>\u6210\u4e3a\u4efb\u4f55\u5934\u90e8\u4f4d\u7f6e\u7684\u952e\u6216\u67e5\u8be2\u7684\u4e24\u4e2a\u7279\u5f81<span translate=no>_^_4_^_</span>\u3002\u6216\u8005\u4e3a\u4e86\u7b80\u5355\u8d77\u89c1<span translate=no>_^_5_^_</span>\uff0c\u5047\u8bbe\u53ea\u6709\u4e24\u4e2a\u529f\u80fd\u3002\u90a3\u4e48\u8f6c\u53d8\u5c31\u662f\uff0c</p>\n<span translate=no>_^_6_^_</span><p>\u5176\u4e2d<span translate=no>_^_7_^_</span>\u662f\u6052\u5b9a\u89d2\u5ea6\u3002\u5176\u4ed6\u8981\u7d20\u5bf9\u7684\u53d8\u6362\u65b9\u5f0f\u7c7b\u4f3c\u3002</p>\n<h3>\u6ce8\u610f\u529b\u662f\u76f8\u5bf9\u7684</h3>\n<p>\u5bf9\u4e8e\u4e00\u5bf9\u529f\u80fd\uff0c\u70b9\u4ea7\u54c1\u6ce8\u610f\u529b\u5206\u6570\u4ecb\u4e8e\u4e24\u4e2a\u4f4d\u7f6e<span translate=no>_^_8_^_</span>\u4e4b\u95f4\uff0c<span translate=no>_^_9_^_</span>\u5c06\u4e3a</p>\n<span translate=no>_^_10_^_</span><p>\u8fd9\u8868\u660e\uff0c\u5bf9\u4e8e\u70b9\u751f\u4ea7\u7684\u5173\u6ce8\uff0c\u65cb\u8f6c\u7f16\u7801\u7ed9\u4e88\u4e86\u76f8\u5bf9\u7684\u5173\u6ce8\u3002</p>\n<h3>\u5bf9\u4e8e\u6240\u6709\u529f\u80fd</h3>\n<p>\u8fd9\u4e9b\u8981\u7d20\u5206\u7ec4\u6210\u5bf9\uff0c\u5e76\u6309\u4e0a\u8ff0\u65b9\u5f0f\u5904\u7406\u3002\u4ed6\u4eec\u5bf9\u6bcf<span translate=no>_^_11_^_</span>\u5bf9\u4f7f\u7528\u4e0d\u540c\u7684\u3002</p>\n<p>\u672c\u6587\u5efa\u8bae\u4f7f\u7528<span translate=no>_^_13_^_</span>\u6210<span translate=no>_^_12_^_</span>\u5bf9\u7684\u7279\u5f81\u3002</p>\n<p>\u6211\u4eec\u5c06\u529f\u80fd<span translate=no>_^_14_^_</span>\u4e0e\u529f\u80fd\u914d\u5bf9<span translate=no>_^_15_^_</span>\u3002\u56e0\u6b64\uff0c\u5bf9\u4e8e\u4f4d\u7f6e<span translate=no>_^_16_^_</span>\u6211\u4eec\u8fdb\u884c\u8f6c\u6362</p>\n<span translate=no>_^_17_^_</span><p>\u81f3</p>\n<span translate=no>_^_18_^_</span>",
|
||||
"<h3>Calculate scores between queries and keys</h3>\n": "<h3>\u8ba1\u7b97\u67e5\u8be2\u548c\u952e\u4e4b\u95f4\u7684\u5206\u6570</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Cache <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> values</p>\n": "<p>\u7f13\u5b58<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span>\u503c</p>\n",
|
||||
"<p> Testing RoPE with a simple example</p>\n": "<p>\u7528\u4e00\u4e2a\u7b80\u5355\u7684\u4f8b\u5b50\u6d4b\u8bd5 RoPe</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Cache <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> values </p>\n": "<p>\u7f13\u5b58<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span>\u503c</p>\n",
|
||||
"<p>Cache them </p>\n": "<p>\u7f13\u5b58\u5b83\u4eec</p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate dot-product with RoPE </p>\n": "<p>\u4f7f\u7528 ROPE \u8ba1\u7b97\u70b9\u79ef</p>\n",
|
||||
"<p>Calculate the product of position index and <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u6301\u4ed3\u6307\u6570\u7684\u4e58\u79ef\u548c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate</p>\n<span translate=no>_^_0_^_</span><p>for <span translate=no>_^_1_^_</span> </p>\n": "<p>\u8ba1\u7b97</p>\n<span translate=no>_^_0_^_</span><p>\u5bf9\u4e8e<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Concatenate so that for row <span translate=no>_^_0_^_</span> we have <span translate=no>_^_1_^_</span> </p>\n": "<p>\u8fde\u63a5\u8fd9\u6837<span translate=no>_^_0_^_</span>\u6211\u4eec\u5c31\u6709 row<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Create position indexes <span translate=no>_^_0_^_</span> </p>\n": "<p>\u521b\u5efa\u5934\u5bf8\u6307\u6570<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get sequence length </p>\n": "<p>\u83b7\u53d6\u5e8f\u5217\u957f\u5ea6</p>\n",
|
||||
"<p>Return if cache is already built </p>\n": "<p>\u5982\u679c\u7f13\u5b58\u5df2\u7ecf\u6784\u5efa\uff0c\u5219\u8fd4\u56de</p>\n",
|
||||
"<p>Rotary positional embedding layers </p>\n": "<p>\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165\u5c42</p>\n",
|
||||
"<p>Split the features, we can choose to apply rotary embeddings only to a partial set of features. </p>\n": "<p>\u62c6\u5206\u7279\u5f81\uff0c\u6211\u4eec\u53ef\u4ee5\u9009\u62e9\u4ec5\u5c06\u65cb\u8f6c\u5d4c\u5165\u5e94\u7528\u4e8e\u90e8\u5206\u7279\u5f81\u96c6\u3002</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the Tensor at the head of a key or a query with shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u4f4d\u4e8e\u952e\u6216\u5e26\u6709\u5f62\u72b6\u7684\u67e5\u8be2\u5f00\u5934\u7684 Tensor<span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the constant used for calculating <span translate=no>_^_3_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8981\u7d20\u7684\u6570\u91cf<span translate=no>_^_1_^_</span></li>\n</ul><li><span translate=no>_^_2_^_</span>\u662f\u7528\u4e8e\u8ba1\u7b97\u7684\u5e38\u6570<span translate=no>_^_3_^_</span></li>\n",
|
||||
"Annotated implementation of RoPE from paper RoFormer: Enhanced Transformer with Rotary Position Embedding": "Paper RoFormer \u4e2d\u5e26\u6ce8\u91ca\u7684 ROPE \u5b9e\u73b0\uff1a\u5e26\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165\u7684\u589e\u5f3a\u578b\u53d8\u538b\u5668",
|
||||
"Rotary Positional Embeddings (RoPE)": "\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (ROPE)"
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings (RoPE) Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a transformer model with Rotary Positional Embeddings (RoPE).</p>\n": "<h1>\u30ed\u30fc\u30bf\u30ea\u30fc\u30fb\u30dd\u30b8\u30b7\u30e7\u30ca\u30eb\u30fb\u30a8\u30f3\u30d9\u30c7\u30a3\u30f3\u30b0 (RoPE) \u5b9f\u9a13</h1>\n<p>\u3053\u308c\u306f\u3001\u56de\u8ee2\u5f0f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\uff08RoPE\uff09\u3092\u4f7f\u7528\u3057\u3066\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304dPyTorch\u5b9f\u9a13\u3067\u3059\u3002</p>\n",
|
||||
"<h3>Rotary PE attention</h3>\n": "<h3>\u30ed\u30fc\u30bf\u30ea\u30fcPE\u6ce8\u610f</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create an autoregressive model and initialize weights</p>\n": "<p>\u81ea\u5df1\u56de\u5e30\u30e2\u30c7\u30eb\u306e\u4f5c\u6210\u3068\u91cd\u307f\u306e\u521d\u671f\u5316</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u8a2d\u5b9a\u30aa\u30d7\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Encoder with RoPE </p>\n": "<p>RoPE \u4ed8\u304d\u30a8\u30f3\u30b3\u30fc\u30c0</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u56fa\u5b9a\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u306a\u3057</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32 \u30a8\u30dd\u30c3\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Noam</a> \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"Rotary Positional Embeddings (RoPE) Experiment": "\u30ed\u30fc\u30bf\u30ea\u30fc\u30fb\u30dd\u30b8\u30b7\u30e7\u30ca\u30eb\u30fb\u30a8\u30f3\u30d9\u30c7\u30a3\u30f3\u30b0 (RoPE) \u5b9f\u9a13",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings (RoPE) on tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u5c0f\u3055\u306a\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u30ed\u30fc\u30bf\u30ea\u30fc\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\uff08RoPE\uff09\u3092\u4f7f\u7528\u3057\u3066\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings (RoPE) Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a transformer model with Rotary Positional Embeddings (RoPE).</p>\n<p><a href=\"https://app.labml.ai/run/1cf508e693be11ecacc98de8b38a61fe\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u0dbb\u0ddc\u0da7\u0dbb\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba\u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0dc3\u0db8\u0d9f \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p><a href=\"https://app.labml.ai/run/1cf508e693be11ecacc98de8b38a61fe\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Rotary PE attention</h3>\n": "<h3>\u0dbb\u0ddc\u0da7\u0dbb\u0dd2PE \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Create an autoregressive model and initialize weights</p>\n": "<p> \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 \u0dc3\u0dc4 \u0db6\u0dbb \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc0\u0dd2\u0d9a\u0dbd\u0dca\u0db4 </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Encoder with RoPE </p>\n": "<p>\u0d9a\u0db9\u0dba\u0dc3\u0db8\u0d9c \u0d86\u0d9a\u0dda\u0dad </p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32\u0dc0\u0dba\u0dc3 \u0d85\u0dc0\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0db1\u0ddd\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"Rotary Positional Embeddings (RoPE) Experiment": "\u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings (RoPE) on tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba\u0dda \u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0dc3\u0dc4\u0dd2\u0dad \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings (RoPE) Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a transformer model with Rotary Positional Embeddings (RoPE).</p>\n": "<h1>\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (RoPE) \u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u65e8\u5728\u4f7f\u7528\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (RoPE) \u8bad\u7ec3\u53d8\u538b\u5668\u6a21\u578b\u3002</p>\n",
|
||||
"<h3>Rotary PE attention</h3>\n": "<h3>Rotary PE \u6ce8\u610f</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Create an autoregressive model and initialize weights</p>\n": "<p>\u521b\u5efa\u81ea\u56de\u5f52\u6a21\u578b\u5e76\u521d\u59cb\u5316\u6743\u91cd</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u914d\u7f6e\u9009\u9879</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Encoder with RoPE </p>\n": "<p>\u5e26\u7ef3\u7684\u7f16\u7801\u5668</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u578b\u53f7\u5c3a\u5bf8</p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u6ca1\u6709\u56fa\u5b9a\u7684\u4f4d\u7f6e\u5d4c\u5165</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>\u8bad\u7ec3 32 \u4e2a\u65f6\u4ee3</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Noam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Noam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"Rotary Positional Embeddings (RoPE) Experiment": "\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (roPE) \u5b9e\u9a8c",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings (RoPE) on tiny Shakespeare dataset.": "\u672c\u5b9e\u9a8c\u5728\u5fae\u5c0f\u7684\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u4e2d\u4f7f\u7528\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165\uff08RoPe\uff09\u8bad\u7ec3\u53d8\u538b\u5668\u6a21\u578b\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings with Relative distance (<a href=\"index.html\">RoPER</a>) Experiment</h1>\n": "<h1><a href=\"index.html\">\u76f8\u5bfe\u8ddd\u96e2\u306b\u3088\u308b\u56de\u8ee2\u5f0f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\uff08RoPer\uff09\u5b9f\u9a13</a></h1>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Use Rotary Positional Embeddings with Relative distance (<a href=\"index.html\">RoPER</a>) in attention.</p>\n": "<p>\u76f8\u5bfe\u8ddd\u96e2 (<a href=\"index.html\">RoPER</a>) \u3092\u8003\u616e\u3057\u305f\u56de\u8ee2\u5f0f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u3092\u4f7f\u7528\u3057\u3066\u304f\u3060\u3055\u3044\u3002</p>\n",
|
||||
"<p> We inherit <a href=\"../experiment.html\">RoPE experiment</a> and use it for <a href=\"../../experiments/arithmetic_dataset.html\">arithmetic addition task</a>.</p>\n<p>We add the option to change attention to use Rotary Positional Embeddings with Relative distance (RoPER) below.</p>\n": "<p><a href=\"../experiment.html\">RoPE\u306e\u5b9f\u9a13\u3092\u7d99\u627f\u3057</a>\u3001<a href=\"../../experiments/arithmetic_dataset.html\">\u7b97\u8853\u52a0\u7b97\u30bf\u30b9\u30af\u306b\u4f7f\u7528\u3057\u307e\u3059</a>\u3002</p>\n<p>\u4ee5\u4e0b\u306b\u6ce8\u76ee\u3092\u5909\u66f4\u3057\u3066\u3001\u76f8\u5bfe\u8ddd\u96e2\u306e\u3042\u308b\u56de\u8ee2\u5f0f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f (RoPer) \u3092\u4f7f\u7528\u3059\u308b\u3088\u3046\u306b\u3057\u307e\u3057\u305f\u3002</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u8a2d\u5b9a\u30aa\u30d7\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Encoder with RoPE attention 'transformer.encoder_attn': 'rotary', </p>\n": "<p>RoPE \u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u4ed8\u304d\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc 'transformer.encoder_attn': '\u30ed\u30fc\u30bf\u30ea\u30fc'\u3001</p>\n",
|
||||
"<p>Encoder with RoPER attention </p>\n": "<p>\u30ed\u30fc\u30d1\u30fc\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u4ed8\u304d\u30a8\u30f3\u30b3\u30fc\u30c0</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u56fa\u5b9a\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u306a\u3057</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32 \u30a8\u30dd\u30c3\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Adam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</a></p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"Rotary Positional Embeddings with Relative distance (RoPER) Experiment": "\u76f8\u5bfe\u8ddd\u96e2\u306b\u3088\u308b\u56de\u8ee2\u5f0f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\uff08RoPer\uff09\u5b9f\u9a13",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings with Relative Distance (RoPER) on the arithmetic addition task.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u7b97\u8853\u52a0\u7b97\u30bf\u30b9\u30af\u3067\u3001\u76f8\u5bfe\u8ddd\u96e2\u4ed8\u304d\u306e\u56de\u8ee2\u5f0f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f (RoPer) \u3092\u4f7f\u7528\u3057\u305f\u5909\u5727\u5668\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings with Relative distance (<a href=\"index.html\">RoPER</a>) Experiment</h1>\n": "<h1>\u0dc3\u0dcf\u0db4\u0dda\u0d9a\u0dca\u0dc2\u0daf\u0dd4\u0dbb (<a href=\"index.html\">ROPER</a>) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dc4\u0dd2\u0dad \u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca</h1>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Use Rotary Positional Embeddings with Relative distance (<a href=\"index.html\">RoPER</a>) in attention.</p>\n": "<p> \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3 \u0dc3\u0dcf\u0db4\u0dda\u0d9a\u0dca\u0dc2 \u0daf\u0dd4\u0dbb (<a href=\"index.html\">ROPER</a>) \u0dc3\u0dc4\u0dd2\u0dad \u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p> We inherit <a href=\"../experiment.html\">RoPE experiment</a> and use it for <a href=\"../../experiments/arithmetic_dataset.html\">arithmetic addition task</a>.</p>\n<p>We add the option to change attention to use Rotary Positional Embeddings with Relative distance (RoPER) below.</p>\n": "<p> \u0d85\u0db4\u0dd2 <a href=\"../experiment.html\">\u0d9a\u0db9\u0dba \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf</a> \u0d8b\u0dbb\u0dd4\u0db8 \u0d9a\u0dbb <a href=\"../../experiments/arithmetic_dataset.html\">\u0d85\u0d82\u0d9a \u0d9c\u0dab\u0dd2\u0dad\u0db8\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba</a> \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4. </p>\n<p>\u0d85\u0db4\u0dd2\u0db4\u0dc4\u0dad \u0dc3\u0dcf\u0db4\u0dda\u0d9a\u0dca\u0dc2 \u0daf\u0dd4\u0dbb (ROPER) \u0dc3\u0db8\u0d9c \u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0dc0\u0dd2\u0d9a\u0dbd\u0dca\u0db4\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc0\u0dd2\u0d9a\u0dbd\u0dca\u0db4 </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Encoder with RoPE attention 'transformer.encoder_attn': 'rotary', </p>\n": "<p>\u0d9a\u0db9\u0dba\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dc4\u0dd2\u0dad \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba 'transformer.encoder_attn': '\u0db7\u0dca\u0dbb\u0db8\u0dab\u0dba', </p>\n",
|
||||
"<p>Encoder with RoPER attention </p>\n": "<p>ROPER\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dc4\u0dd2\u0dad \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba </p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32\u0dc0\u0dba\u0dc3 \u0d85\u0dc0\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Adam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"Rotary Positional Embeddings with Relative distance (RoPER) Experiment": "\u0dc3\u0dcf\u0db4\u0dda\u0d9a\u0dca\u0dc2 \u0daf\u0dd4\u0dbb (ROPER) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dc4\u0dd2\u0dad \u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings with Relative Distance (RoPER) on the arithmetic addition task.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d85\u0d82\u0d9a \u0d9c\u0dab\u0dd2\u0dad\u0db8\u0dba \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0dba \u0db8\u0dad \u0dc3\u0dcf\u0db4\u0dda\u0d9a\u0dca\u0dc2 \u0daf\u0dd4\u0dbb (ROPER) \u0dc3\u0dc4\u0dd2\u0dad \u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0dc4\u0dd2\u0dad \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings with Relative distance (<a href=\"index.html\">RoPER</a>) Experiment</h1>\n": "<h1>\u76f8\u5bf9\u8ddd\u79bb\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (<a href=\"index.html\">RoPer</a>) \u5b9e\u9a8c</h1>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Use Rotary Positional Embeddings with Relative distance (<a href=\"index.html\">RoPER</a>) in attention.</p>\n": "<p>\u6ce8\u610f\u4f7f\u7528\u5177\u6709\u76f8\u5bf9\u8ddd\u79bb\u7684\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (<a href=\"index.html\">RoPer</a>)\u3002</p>\n",
|
||||
"<p> We inherit <a href=\"../experiment.html\">RoPE experiment</a> and use it for <a href=\"../../experiments/arithmetic_dataset.html\">arithmetic addition task</a>.</p>\n<p>We add the option to change attention to use Rotary Positional Embeddings with Relative distance (RoPER) below.</p>\n": "<p>\u6211\u4eec\u7ee7\u627f\u4e86 <a href=\"../experiment.html\">RoPe \u5b9e\u9a8c</a>\uff0c\u5e76\u5c06\u5176\u7528\u4e8e<a href=\"../../experiments/arithmetic_dataset.html\">\u7b97\u672f\u52a0\u6cd5\u4efb\u52a1</a>\u3002</p>\n<p>\u6211\u4eec\u5728\u4e0b\u9762\u6dfb\u52a0\u4e86\u66f4\u6539\u6ce8\u610f\u529b\u7684\u9009\u9879\uff0c\u4ee5\u4f7f\u7528\u76f8\u5bf9\u8ddd\u79bb\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (RoPer)\u3002</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u914d\u7f6e\u9009\u9879</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Encoder with RoPE attention 'transformer.encoder_attn': 'rotary', </p>\n": "<p>\u5e26\u6709 roPe \u6ce8\u610f\u529b\u7684\u7f16\u7801\u5668 \u201ctransformer.encoder_attn\u201d\uff1a\u201crotary\u201d\uff0c</p>\n",
|
||||
"<p>Encoder with RoPER attention </p>\n": "<p>RoPer \u5173\u6ce8\u7684\u7f16\u7801\u5668</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u578b\u53f7\u5c3a\u5bf8</p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u6ca1\u6709\u56fa\u5b9a\u7684\u4f4d\u7f6e\u5d4c\u5165</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>\u8bad\u7ec3 32 \u4e2a\u65f6\u4ee3</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Adam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Adam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"Rotary Positional Embeddings with Relative distance (RoPER) Experiment": "\u76f8\u5bf9\u8ddd\u79bb\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (RoPer) \u5b9e\u9a8c",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings with Relative Distance (RoPER) on the arithmetic addition task.": "\u672c\u5b9e\u9a8c\u5728\u7b97\u672f\u52a0\u6cd5\u4efb\u52a1\u4e2d\u4f7f\u7528\u76f8\u5bf9\u8ddd\u79bb\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165\uff08RoPer\uff09\u8bad\u7ec3\u53d8\u538b\u5668\u6a21\u578b\u3002"
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings (RoPE) Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a transformer model with Rotary Positional Embeddings (RoPE).</p>\n": "<h1>\u30ed\u30fc\u30bf\u30ea\u30fc\u30fb\u30dd\u30b8\u30b7\u30e7\u30ca\u30eb\u30fb\u30a8\u30f3\u30d9\u30c7\u30a3\u30f3\u30b0 (RoPE) \u5b9f\u9a13</h1>\n<p>\u3053\u308c\u306f\u3001\u56de\u8ee2\u5f0f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\uff08RoPE\uff09\u3092\u4f7f\u7528\u3057\u3066\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304dPyTorch\u5b9f\u9a13\u3067\u3059\u3002</p>\n",
|
||||
"<h3>Rotary PE attention</h3>\n": "<h3>\u30ed\u30fc\u30bf\u30ea\u30fcPE\u6ce8\u610f</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>'transformer.encoder_attn': 'rotary', </p>\n": "<p>'\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc.encoder_attn ':' \u30ed\u30fc\u30bf\u30ea\u30fc '\u3001</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u8a2d\u5b9a\u30aa\u30d7\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Encoder with RoPE </p>\n": "<p>RoPE \u4ed8\u304d\u30a8\u30f3\u30b3\u30fc\u30c0</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u30e2\u30c7\u30eb\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u56fa\u5b9a\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u306a\u3057</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32 \u30a8\u30dd\u30c3\u30af\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Adam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">Adam \u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</a></p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
|
||||
"Rotary Positional Embeddings (RoPE) Experiment": "\u30ed\u30fc\u30bf\u30ea\u30fc\u30fb\u30dd\u30b8\u30b7\u30e7\u30ca\u30eb\u30fb\u30a8\u30f3\u30d9\u30c7\u30a3\u30f3\u30b0 (RoPE) \u5b9f\u9a13",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings (RoPE) on tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u5c0f\u3055\u306a\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u30ed\u30fc\u30bf\u30ea\u30fc\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\uff08RoPE\uff09\u3092\u4f7f\u7528\u3057\u3066\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings (RoPE) Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a transformer model with Rotary Positional Embeddings (RoPE).</p>\n<p><a href=\"https://app.labml.ai/run/1cf508e693be11ecacc98de8b38a61fe\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u0dbb\u0ddc\u0da7\u0dbb\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0dba\u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0dc3\u0db8\u0d9f \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p><a href=\"https://app.labml.ai/run/1cf508e693be11ecacc98de8b38a61fe\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h3>Rotary PE attention</h3>\n": "<h3>\u0dbb\u0ddc\u0da7\u0dbb\u0dd2PE \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>'transformer.encoder_attn': 'rotary', </p>\n": "<p>'\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca.encoder_attn':' \u0db7\u0dca\u0dbb\u0db8\u0d9a ', </p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc0\u0dd2\u0d9a\u0dbd\u0dca\u0db4 </p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Encoder with RoPE </p>\n": "<p>\u0d9a\u0db9\u0dba\u0dc3\u0db8\u0d9c \u0d86\u0d9a\u0dda\u0dad </p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db1\u0ddc\u0db8\u0dd0\u0dad </p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>32\u0dc0\u0dba\u0dc3 \u0d85\u0dc0\u0dd4\u0dbb\u0dd4\u0daf\u0dd4 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Adam optimizer</a> </p>\n": "<p><a href=\"../../optimizers/noam.html\">\u0d86\u0daf\u0db8\u0dca \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf </p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"Rotary Positional Embeddings (RoPE) Experiment": "\u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings (RoPE) on tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba\u0dda \u0dbb\u0ddc\u0da7\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca (\u0d9a\u0db9\u0dba) \u0dc3\u0dc4\u0dd2\u0dad \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"<h1>Rotary Positional Embeddings (RoPE) Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a transformer model with Rotary Positional Embeddings (RoPE).</p>\n": "<h1>\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (RoPE) \u5b9e\u9a8c</h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u65e8\u5728\u4f7f\u7528\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (RoPE) \u8bad\u7ec3\u53d8\u538b\u5668\u6a21\u578b\u3002</p>\n",
|
||||
"<h3>Rotary PE attention</h3>\n": "<h3>Rotary PE \u6ce8\u610f</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>'transformer.encoder_attn': 'rotary', </p>\n": "<p>'transformer.encoder_attn '\uff1a' rotary '\uff0c</p>\n",
|
||||
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Configuration options </p>\n": "<p>\u914d\u7f6e\u9009\u9879</p>\n",
|
||||
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Encoder with RoPE </p>\n": "<p>\u5e26\u7ef3\u7684\u7f16\u7801\u5668</p>\n",
|
||||
"<p>Model size </p>\n": "<p>\u578b\u53f7\u5c3a\u5bf8</p>\n",
|
||||
"<p>No fixed positional embeddings </p>\n": "<p>\u6ca1\u6709\u56fa\u5b9a\u7684\u4f4d\u7f6e\u5d4c\u5165</p>\n",
|
||||
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
|
||||
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
|
||||
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
|
||||
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
|
||||
"<p>Train for 32 epochs </p>\n": "<p>\u8bad\u7ec3 32 \u4e2a\u65f6\u4ee3</p>\n",
|
||||
"<p>Use <a href=\"../../optimizers/noam.html\">Adam optimizer</a> </p>\n": "<p>\u4f7f\u7528 <a href=\"../../optimizers/noam.html\">Adam \u4f18\u5316\u5668</a></p>\n",
|
||||
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
|
||||
"Rotary Positional Embeddings (RoPE) Experiment": "\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165 (roPE) \u5b9e\u9a8c",
|
||||
"This experiment trains a transformer model with Rotary Positional Embeddings (RoPE) on tiny Shakespeare dataset.": "\u672c\u5b9e\u9a8c\u5728\u5fae\u5c0f\u7684\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u4e2d\u4f7f\u7528\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165\uff08RoPe\uff09\u8bad\u7ec3\u53d8\u538b\u5668\u6a21\u578b\u3002"
|
||||
}
|
||||
Reference in New Issue
Block a user