chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Retrieval-Enhanced Transformer (Retro)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2112.04426\">Improving language models by retrieving from trillions of tokens</a>.</p>\n<p>It builds a database of chunks of text. It is a key-value database where the keys are indexed by the BERT embeddings of the chunks. They use a frozen pre-trained BERT model to calculate these embeddings. The values are the corresponding chunks and an equal length of text proceeding that chunk.</p>\n<p>Then the model retrieves text similar (nearest neighbors) to the input to the model from this database. These retrieved texts are used to predict the output.</p>\n<p>Since we use a frozen BERT model for retrieval we can pre-calculate all the nearest neighbors for the training dataset. This speeds up the training process.</p>\n<p>Components:</p>\n<ul><li><a href=\"bert_embeddings.html\">BERT embeddings</a>: Code to get BERT embeddings of chunks of text. </li>\n<li><a href=\"database.html\">Key-value database</a>: Build and retrieve chunks </li>\n<li><a href=\"model.html\">Model</a> </li>\n<li><a href=\"dataset.html\">Dataset</a>: Pre-calculate the nearest neighbors </li>\n<li><a href=\"train.html\">Training code</a></li></ul>\n": "<h1>\u691c\u7d22\u6a5f\u80fd\u4ed8\u304d\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc (\u30ec\u30c8\u30ed)</h1>\n<p>\u3053\u308c\u306f\u3001<a href=\"https://pytorch.org\"><a href=\"https://arxiv.org/abs/2112.04426\">\u6570\u5146\u306e\u30c8\u30fc\u30af\u30f3\u304b\u3089\u53d6\u5f97\u3059\u308b\u3053\u3068\u306b\u3088\u308b\u8a00\u8a9e\u30e2\u30c7\u30eb\u306e\u6539\u5584\u3068\u3044\u3046\u8ad6\u6587\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n<p>\u30c6\u30ad\u30b9\u30c8\u306e\u584a\u306e\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u3092\u69cb\u7bc9\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u30c1\u30e3\u30f3\u30af\u306eBERT\u57cb\u3081\u8fbc\u307f\u306b\u3088\u3063\u3066\u30ad\u30fc\u306e\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u304c\u4ed8\u3051\u3089\u308c\u308b\u30ad\u30fc\u30d0\u30ea\u30e5\u30fc\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u3067\u3059\u3002\u3053\u308c\u3089\u306e\u57cb\u3081\u8fbc\u307f\u3092\u8a08\u7b97\u3059\u308b\u306b\u306f\u3001\u4e8b\u524d\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u6e08\u307f\u306e\u30d5\u30ea\u30fc\u30ba\u3057\u305f BERT \u30e2\u30c7\u30eb\u3092\u4f7f\u7528\u3057\u307e\u3059\u3002\u5024\u306f\u3001\u5bfe\u5fdc\u3059\u308b\u30c1\u30e3\u30f3\u30af\u3068\u3001\u305d\u306e\u30c1\u30e3\u30f3\u30af\u306e\u524d\u306b\u7d9a\u304f\u540c\u3058\u9577\u3055\u306e\u30c6\u30ad\u30b9\u30c8\u3067\u3059\u3002</p>\n<p>\u6b21\u306b\u3001\u30e2\u30c7\u30eb\u306f\u3053\u306e\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u304b\u3089\u30e2\u30c7\u30eb\u3078\u306e\u5165\u529b\u306b\u985e\u4f3c\u3057\u305f (\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u306e) \u30c6\u30ad\u30b9\u30c8\u3092\u53d6\u5f97\u3057\u307e\u3059\u3002\u53d6\u5f97\u3057\u305f\u3053\u308c\u3089\u306e\u30c6\u30ad\u30b9\u30c8\u306f\u3001\u51fa\u529b\u306e\u4e88\u6e2c\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059</p>\u3002\n<p>\u691c\u7d22\u306b\u306f\u30d5\u30ea\u30fc\u30ba\u3057\u305f BERT \u30e2\u30c7\u30eb\u3092\u4f7f\u7528\u3059\u308b\u305f\u3081\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u3059\u3079\u3066\u306e\u6700\u8fd1\u508d\u3092\u4e8b\u524d\u306b\u8a08\u7b97\u3067\u304d\u307e\u3059\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30d7\u30ed\u30bb\u30b9\u304c\u30b9\u30d4\u30fc\u30c9\u30a2\u30c3\u30d7\u3057\u307e\u3059\u3002</p>\n<p>\u30b3\u30f3\u30dd\u30fc\u30cd\u30f3\u30c8:</p>\n<ul><li><a href=\"bert_embeddings.html\">BERT \u57cb\u3081\u8fbc\u307f:\u30c6\u30ad\u30b9\u30c8\u306e\u30c1\u30e3\u30f3\u30af\u3092</a> BERT \u306b\u57cb\u3081\u8fbc\u3080\u305f\u3081\u306e\u30b3\u30fc\u30c9\u3002</li>\n<li><a href=\"database.html\">\u30ad\u30fc\u30d0\u30ea\u30e5\u30fc\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9:\u30c1\u30e3\u30f3\u30af\u306e\u69cb\u7bc9\u3068\u53d6\u5f97</a></li>\n<li><a href=\"model.html\">\u30e2\u30c7\u30eb</a></li>\n<li><a href=\"dataset.html\">\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8</a>:\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u3092\u4e8b\u524d\u8a08\u7b97</li>\n<li><a href=\"train.html\">\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9</a></li></ul>\n",
|
||||
"Retrieval-Enhanced Transformer (Retro)": "\u691c\u7d22\u6a5f\u80fd\u4ed8\u304d\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc (\u30ec\u30c8\u30ed)",
|
||||
"This is a PyTorch implementation/tutorial of the paper Improving language models by retrieving from trillions of tokens. It builds a key-value database of chunks of text and retrieves and uses them when making predictions.": "\u3053\u308c\u306f\u3001\u6570\u5146\u306e\u30c8\u30fc\u30af\u30f3\u304b\u3089\u53d6\u5f97\u3059\u308b\u3053\u3068\u306b\u3088\u308b\u8a00\u8a9e\u30e2\u30c7\u30eb\u306e\u6539\u5584\u3068\u3044\u3046\u8ad6\u6587\u306ePyTorch\u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3067\u3059\u3002\u30c6\u30ad\u30b9\u30c8\u306e\u30c1\u30e3\u30f3\u30af\u304b\u3089\u6210\u308b\u30ad\u30fc\u30d0\u30ea\u30e5\u30fc\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u3092\u69cb\u7bc9\u3057\u3001\u305d\u308c\u3089\u3092\u53d6\u5f97\u3057\u3066\u4e88\u6e2c\u306b\u4f7f\u7528\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<h1>Retrieval-Enhanced Transformer (Retro)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2112.04426\">Improving language models by retrieving from trillions of tokens</a>.</p>\n<p>It builds a database of chunks of text. It is a key-value database where the keys are indexed by the BERT embeddings of the chunks. They use a frozen pre-trained BERT model to calculate these embeddings. The values are the corresponding chunks and an equal length of text proceeding that chunk.</p>\n<p>Then the model retrieves text similar (nearest neighbors) to the input to the model from this database. These retrieved texts are used to predict the output.</p>\n<p>Since we use a frozen BERT model for retrieval we can pre-calculate all the nearest neighbors for the training dataset. This speeds up the training process.</p>\n<p>Components:</p>\n<ul><li><a href=\"bert_embeddings.html\">BERT embeddings</a>: Code to get BERT embeddings of chunks of text. </li>\n<li><a href=\"database.html\">Key-value database</a>: Build and retrieve chunks </li>\n<li><a href=\"model.html\">Model</a> </li>\n<li><a href=\"dataset.html\">Dataset</a>: Pre-calculate the nearest neighbors </li>\n<li><a href=\"train.html\">Training code</a></li></ul>\n": "<h1>\u68c0\u7d22\u589e\u5f3a\u578b\u53d8\u538b\u5668\uff08\u590d\u53e4\uff09</h1>\n<p>\u8fd9\u662f <a href=\"https://pytorch.org\">PyTorch</a> \u5bf9\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/2112.04426\">\u901a\u8fc7\u4ece\u6570\u4e07\u4ebf\u4e2a\u4ee3\u5e01\u4e2d\u68c0\u7d22\u6765\u6539\u8fdb\u8bed\u8a00\u6a21\u578b\u300b\u7684</a>\u5b9e\u73b0\u3002</p>\n<p>\u5b83\u5efa\u7acb\u4e86\u4e00\u4e2a\u5305\u542b\u5927\u91cf\u6587\u672c\u7684\u6570\u636e\u5e93\u3002\u5b83\u662f\u4e00\u4e2a\u952e\u503c\u6570\u636e\u5e93\uff0c\u5176\u4e2d\u7684\u5bc6\u94a5\u7531\u533a\u5757\u7684 BERT \u5d4c\u5165\u7d22\u5f15\u3002\u4ed6\u4eec\u4f7f\u7528\u51bb\u7ed3\u7684\u9884\u8bad\u7ec3\u7684 BERT \u6a21\u578b\u6765\u8ba1\u7b97\u8fd9\u4e9b\u5d4c\u5165\u3002\u8fd9\u4e9b\u503c\u662f\u76f8\u5e94\u7684\u533a\u5757\u548c\u8be5\u533a\u5757\u7684\u7b49\u957f\u5ea6\u6587\u672c\u3002</p>\n<p>\u7136\u540e\uff0c\u6a21\u578b\u4ece\u8be5\u6570\u636e\u5e93\u68c0\u7d22\u4e0e\u6a21\u578b\u8f93\u5165\u76f8\u4f3c\uff08\u6700\u8fd1\u90bb\u57df\uff09\u7684\u6587\u672c\u3002\u8fd9\u4e9b\u68c0\u7d22\u5230\u7684\u6587\u672c\u7528\u4e8e\u9884\u6d4b\u8f93\u51fa\u3002</p>\n<p>\u7531\u4e8e\u6211\u4eec\u4f7f\u7528\u51bb\u7ed3\u7684 BERT \u6a21\u578b\u8fdb\u884c\u68c0\u7d22\uff0c\u56e0\u6b64\u6211\u4eec\u53ef\u4ee5\u9884\u5148\u8ba1\u7b97\u8bad\u7ec3\u6570\u636e\u96c6\u7684\u6240\u6709\u6700\u8fd1\u90bb\u57df\u3002\u8fd9\u52a0\u5feb\u4e86\u8bad\u7ec3\u8fc7\u7a0b\u3002</p>\n<p>\u7ec4\u4ef6\uff1a</p>\n<ul><li><a href=\"bert_embeddings.html\">BERT \u5d4c\u5165</a>\uff1a\u7528\u4e8e\u83b7\u53d6\u5927\u5757\u6587\u672c\u7684 BERT \u5d4c\u5165\u7684\u4ee3\u7801\u3002</li>\n<li><a href=\"database.html\">\u952e\u503c\u6570\u636e\u5e93</a>\uff1a\u751f\u6210\u548c\u68c0\u7d22\u533a\u5757</li>\n<li><a href=\"model.html\">\u6a21\u578b</a></li>\n<li><a href=\"dataset.html\">\u6570\u636e\u96c6</a>\uff1a\u9884\u5148\u8ba1\u7b97\u6700\u8fd1\u7684\u90bb\u5c45</li>\n<li><a href=\"train.html\">\u8bad\u7ec3\u4ee3\u7801</a></li></ul>\n",
|
||||
"Retrieval-Enhanced Transformer (Retro)": "\u68c0\u7d22\u589e\u5f3a\u578b\u53d8\u538b\u5668\uff08\u590d\u53e4\uff09",
|
||||
"This is a PyTorch implementation/tutorial of the paper Improving language models by retrieving from trillions of tokens. It builds a key-value database of chunks of text and retrieves and uses them when making predictions.": "\u8fd9\u662f\u8bba\u6587\u300a\u901a\u8fc7\u4ece\u6570\u4e07\u4ebf\u4e2a\u4ee4\u724c\u4e2d\u68c0\u7d22\u6539\u8fdb\u8bed\u8a00\u6a21\u578b\u300b\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002\u5b83\u6784\u5efa\u4e86\u4e00\u4e2a\u5305\u542b\u6587\u672c\u5757\u7684\u952e\u503c\u6570\u636e\u5e93\uff0c\u5e76\u5728\u8fdb\u884c\u9884\u6d4b\u65f6\u68c0\u7d22\u548c\u4f7f\u7528\u5b83\u4eec\u3002"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1>BERT Embeddings of chunks of text</h1>\n<p>This is the code to get BERT embeddings of chunks for <a href=\"index.html\">RETRO model</a>.</p>\n": "<h1>BERT \u30c6\u30ad\u30b9\u30c8\u306e\u584a\u306e\u57cb\u3081\u8fbc\u307f</h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f\u3001RETRO\u30e2\u30c7\u30eb\u7528\u306e\u30c1\u30e3\u30f3\u30af\u306eBERT\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97\u3059\u308b\u305f\u3081\u306e\u30b3\u30fc\u30c9\u3067\u3059\u3002</a></p>\n",
|
||||
"<h2>BERT Embeddings</h2>\n<p>For a given chunk of text <span translate=no>_^_0_^_</span> this class generates BERT embeddings <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> is the average of BERT embeddings of all the tokens in <span translate=no>_^_3_^_</span>.</p>\n": "<h2>BERT \u30a8\u30f3\u30d9\u30c7\u30a3\u30f3\u30b0</h2>\n<p><span translate=no>_^_0_^_</span>\u3053\u306e\u30af\u30e9\u30b9\u306f\u3001\u7279\u5b9a\u306e\u30c6\u30ad\u30b9\u30c8\u30c1\u30e3\u30f3\u30af\u306b\u5bfe\u3057\u3066 BERT <span translate=no>_^_1_^_</span> \u57cb\u3081\u8fbc\u307f\u3092\u751f\u6210\u3057\u307e\u3059\u3002<span translate=no>_^_2_^_</span>\u306f\u3001\u5185\u306e\u3059\u3079\u3066\u306e\u30c8\u30fc\u30af\u30f3\u306eBERT\u57cb\u3081\u8fbc\u307f\u306e\u5e73\u5747\u3067\u3059\u3002<span translate=no>_^_3_^_</span></p>\n",
|
||||
"<h3>Code to test BERT embeddings</h3>\n": "<h3>BERT \u57cb\u3081\u8fbc\u307f\u3092\u30c6\u30b9\u30c8\u3059\u308b\u30b3\u30fc\u30c9</h3>\n",
|
||||
"<h3>Get <span translate=no>_^_0_^_</span> for a list of chunks.</h3>\n": "<h3><span translate=no>_^_0_^_</span>\u30c1\u30e3\u30f3\u30af\u306e\u30ea\u30b9\u30c8\u3092\u5165\u624b\u3057\u3066\u304f\u3060\u3055\u3044\u3002</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> In this implementation, we do not make chunks with a fixed number of tokens. One of the reasons is that this implementation uses character-level tokens and BERT uses its sub-word tokenizer.</p>\n<p>So this method will truncate the text to make sure there are no partial tokens.</p>\n<p>For instance, a chunk could be like <span translate=no>_^_0_^_</span>, with partial words (partial sub-word tokens) on the ends. We strip them off to get better BERT embeddings. As mentioned earlier this is not necessary if we broke chunks after tokenizing.</p>\n": "<p>\u3053\u306e\u5b9f\u88c5\u3067\u306f\u3001\u56fa\u5b9a\u6570\u306e\u30c8\u30fc\u30af\u30f3\u3067\u30c1\u30e3\u30f3\u30af\u3092\u4f5c\u6210\u3057\u307e\u305b\u3093\u3002\u7406\u7531\u306e1\u3064\u306f\u3001\u3053\u306e\u5b9f\u88c5\u3067\u306f\u6587\u5b57\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30f3\u3092\u4f7f\u7528\u3057\u3001BERT\u306f\u30b5\u30d6\u30ef\u30fc\u30c9\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u7528\u3057\u3066\u3044\u308b\u3053\u3068\u3067\u3059</p>\u3002\n<p>\u305d\u306e\u305f\u3081\u3001\u3053\u306e\u30e1\u30bd\u30c3\u30c9\u306f\u30c6\u30ad\u30b9\u30c8\u3092\u5207\u308a\u6368\u3066\u3066\u3001\u90e8\u5206\u7684\u306a\u30c8\u30fc\u30af\u30f3\u304c\u306a\u3044\u3053\u3068\u3092\u78ba\u8a8d\u3057\u307e\u3059\u3002</p>\n<p>\u305f\u3068\u3048\u3070\u3001\u30c1\u30e3\u30f3\u30af\u306f\u3001\u672b\u5c3e\u306b\u5358\u8a9e\u306e\u4e00\u90e8\uff08\u90e8\u5206\u7684\u306a\u30b5\u30d6\u30ef\u30fc\u30c9\u30c8\u30fc\u30af\u30f3\uff09<span translate=no>_^_0_^_</span>\u304c\u3042\u308b\u3088\u3046\u306a\u3082\u306e\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u3002BERT \u57cb\u3081\u8fbc\u307f\u306e\u7cbe\u5ea6\u3092\u9ad8\u3081\u308b\u305f\u3081\u3001\u3053\u308c\u3089\u3092\u524a\u9664\u3057\u307e\u3057\u305f\u3002\u5148\u306b\u8ff0\u3079\u305f\u3088\u3046\u306b\u3001\u30c8\u30fc\u30af\u30f3\u5316\u5f8c\u306b\u30c1\u30e3\u30f3\u30af\u3092\u5206\u5272\u3057\u305f\u5834\u5408\u306f\u5fc5\u8981\u3042\u308a\u307e\u305b\u3093</p>\u3002\n",
|
||||
"<p>Break words </p>\n": "<p>\u30d6\u30ec\u30fc\u30af\u30fb\u30ef\u30fc\u30c9</p>\n",
|
||||
"<p>Calculate the average token embeddings. Note that the attention mask is <span translate=no>_^_0_^_</span> if the token is empty padded. We get empty tokens because the chunks are of different lengths. </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u306e\u5e73\u5747\u56de\u6570\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002\u6ce8\u610f\u30de\u30b9\u30af\u306f\u3001<span translate=no>_^_0_^_</span>\u30c8\u30fc\u30af\u30f3\u304c\u7a7a\u3067\u30d1\u30c7\u30a3\u30f3\u30b0\u3055\u308c\u3066\u3044\u308b\u5834\u5408\u3067\u3042\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002\u30c1\u30e3\u30f3\u30af\u306e\u9577\u3055\u304c\u7570\u306a\u308b\u305f\u3081\u3001\u7a7a\u306e\u30c8\u30fc\u30af\u30f3\u304c\u8fd4\u3055\u308c\u307e\u3059</p>\u3002\n",
|
||||
"<p>Check BERT model outputs </p>\n": "<p>BERT \u30e2\u30c7\u30eb\u306e\u51fa\u529b\u3092\u30c1\u30a7\u30c3\u30af</p>\n",
|
||||
"<p>Check BERT tokenizer </p>\n": "<p>BERT \u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u30c1\u30a7\u30c3\u30af</p>\n",
|
||||
"<p>Check recreating text from token ids </p>\n": "<p>\u30c8\u30fc\u30af\u30f3 ID \u304b\u3089\u30c6\u30ad\u30b9\u30c8\u3092\u518d\u4f5c\u6210\u3059\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Evaluate the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u8a55\u4fa1</p>\n",
|
||||
"<p>Get chunk embeddings </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the token embeddings </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3092\u5165\u624b</p>\n",
|
||||
"<p>If empty return original string </p>\n": "<p>\u7a7a\u306e\u5834\u5408\u3001\u5143\u306e\u6587\u5b57\u5217\u3092\u8fd4\u3059</p>\n",
|
||||
"<p>Initialize </p>\n": "<p>[\u521d\u671f\u5316]</p>\n",
|
||||
"<p>Load the BERT model from <a href=\"https://huggingface.co/bert-base-uncased\">HuggingFace</a> </p>\n": "<p><a href=\"https://huggingface.co/bert-base-uncased\">\u30cf\u30ae\u30f3\u30b0\u30d5\u30a7\u30a4\u30b9\u304b\u3089</a> BERT \u30e2\u30c7\u30eb\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Load the BERT tokenizer from <a href=\"https://huggingface.co/bert-base-uncased\">HuggingFace</a> </p>\n": "<p><a href=\"https://huggingface.co/bert-base-uncased\">HuggingFace \u304b\u3089 BERT \u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u30ed\u30fc\u30c9\u3057\u307e\u3059</a></p>\n",
|
||||
"<p>Move the model to <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u6b21\u306e\u5834\u6240\u306b\u79fb\u52d5 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Move token ids, attention mask and token types to the device </p>\n": "<p>\u30c8\u30fc\u30af\u30f3 ID\u3001\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30de\u30b9\u30af\u3001\u30c8\u30fc\u30af\u30f3\u30bf\u30a4\u30d7\u3092\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Otherwise, return the stripped string </p>\n": "<p>\u305d\u308c\u4ee5\u5916\u306e\u5834\u5408\u306f\u3001\u53d6\u308a\u9664\u3044\u305f\u6587\u5b57\u5217\u3092\u8fd4\u3059</p>\n",
|
||||
"<p>Remove first and last pieces </p>\n": "<p>\u6700\u521d\u3068\u6700\u5f8c\u306e\u30d4\u30fc\u30b9\u3092\u524a\u9664\u3059\u308b</p>\n",
|
||||
"<p>Remove whitespace </p>\n": "<p>\u7a7a\u767d\u3092\u524a\u9664</p>\n",
|
||||
"<p>Sample </p>\n": "<p>[\u30b5\u30f3\u30d7\u30eb]</p>\n",
|
||||
"<p>Strip whitespace </p>\n": "<p>\u7a7a\u767d\u3092\u524a\u9664</p>\n",
|
||||
"<p>Tokenize the chunks with BERT tokenizer </p>\n": "<p>BERT \u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3067\u30c1\u30e3\u30f3\u30af\u3092\u30c8\u30fc\u30af\u30f3\u5316\u3059\u308b</p>\n",
|
||||
"<p>Trim the chunks </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u3092\u30c8\u30ea\u30df\u30f3\u30b0</p>\n",
|
||||
"<p>We don't need to compute gradients </p>\n": "<p>\u52fe\u914d\u3092\u8a08\u7b97\u3059\u308b\u5fc5\u8981\u306f\u3042\u308a\u307e\u305b\u3093</p>\n",
|
||||
"BERT Embeddings of chunks of text": "BERT \u30c6\u30ad\u30b9\u30c8\u306e\u584a\u306e\u57cb\u3081\u8fbc\u307f",
|
||||
"Generate BERT embeddings for chunks using a frozen BERT model": "\u30d5\u30ea\u30fc\u30ba\u3057\u305f BERT \u30e2\u30c7\u30eb\u3092\u4f7f\u7528\u3057\u3066\u30c1\u30e3\u30f3\u30af\u306e BERT \u57cb\u3081\u8fbc\u307f\u3092\u751f\u6210"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1>BERT Embeddings of chunks of text</h1>\n<p>This is the code to get BERT embeddings of chunks for <a href=\"index.html\">RETRO model</a>.</p>\n": "<h1>\u0db4\u0dd9\u0dc5\u0d9a\u0dd0\u0db6\u0dbd\u0dd2 \u0dc0\u0dbd \u0db6\u0dbb\u0dca\u0da7\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca</h1>\n<p><a href=\"index.html\">RETRO \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</a>\u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc0\u0dbd BERT \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0dda \u0d9a\u0dda\u0dad\u0dba \u0db8\u0dd9\u0dba\u0dba\u0dd2. </p>\n",
|
||||
"<h2>BERT Embeddings</h2>\n<p>For a given chunk of text <span translate=no>_^_0_^_</span> this class generates BERT embeddings <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> is the average of BERT embeddings of all the tokens in <span translate=no>_^_3_^_</span>.</p>\n": "<h2>\u0db6\u0dbb\u0dca\u0da7\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca</h2>\n<p>\u0daf\u0dd3\u0d87\u0dad\u0dd2 \u0db4\u0dd9\u0dc5 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0db3\u0dc4\u0dcf <span translate=no>_^_0_^_</span> \u0db8\u0dd9\u0db8 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba BERT \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2 <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> \u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8 \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dbd BERT \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd <span translate=no>_^_3_^_</span>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba \u0dc0\u0dda. </p>\n",
|
||||
"<h3>Code to test BERT embeddings</h3>\n": "<h3>BERT\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d9a\u0dda\u0dad\u0dba</h3>\n",
|
||||
"<h3>Get <span translate=no>_^_0_^_</span> for a list of chunks.</h3>\n": "<h3>\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1. </h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> In this implementation, we do not make chunks with a fixed number of tokens. One of the reasons is that this implementation uses character-level tokens and BERT uses its sub-word tokenizer.</p>\n<p>So this method will truncate the text to make sure there are no partial tokens.</p>\n<p>For instance, a chunk could be like <span translate=no>_^_0_^_</span>, with partial words (partial sub-word tokens) on the ends. We strip them off to get better BERT embeddings. As mentioned earlier this is not necessary if we broke chunks after tokenizing.</p>\n": "<p> \u0db8\u0dd9\u0db8\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3, \u0d85\u0db4\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0dc0\u0dbb \u0da7\u0ddd\u0d9a\u0db1 \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0\u0d9a\u0dca \u0dc3\u0db8\u0d9f \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1\u0dda \u0db1\u0dd0\u0dad. \u0d91\u0d9a\u0dca \u0dc4\u0dda\u0dad\u0dd4\u0dc0\u0d9a\u0dca \u0db1\u0db8\u0dca, \u0db8\u0dd9\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0da0\u0dbb\u0dd2\u0dad \u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 BERT \u0d91\u0dc4\u0dd2 \u0d8b\u0db4 \u0dc0\u0da0\u0db1 \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dba\u0dd2. </p>\n<p>\u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca\u0db8\u0dd9\u0db8 \u0d9a\u0dca\u0dbb\u0db8\u0dba \u0db8\u0d9f\u0dd2\u0db1\u0dca \u0d85\u0dbb\u0dca\u0db0 \u0da7\u0ddd\u0d9a\u0db1 \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db6\u0dc0\u0da7 \u0dc0\u0d9c \u0db6\u0dbd\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dcf truncate \u0dc0\u0dda. </p>\n<p>\u0db1\u0dd2\u0daf\u0dc3\u0dd4\u0db1\u0d9a\u0dca\u0dc0\u0dc1\u0dba\u0dd9\u0db1\u0dca, \u0d9a\u0dd9\u0dc5\u0dc0\u0dbb\u0dda \u0d85\u0dbb\u0dca\u0db0 \u0dc0\u0da0\u0db1 (\u0d85\u0dbb\u0dca\u0db0 \u0d8b\u0db4 \u0dc0\u0da0\u0db1 \u0da7\u0ddd\u0d9a\u0db1) \u0dc3\u0dc4\u0dd2\u0dad \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0db8\u0dcf\u0db1 <span translate=no>_^_0_^_</span>\u0dc0\u0dd2\u0dba \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. \u0dc0\u0da9\u0dcf \u0dc4\u0ddc\u0db3 \u0db6\u0dbb\u0dca\u0da7\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 \u0d92\u0dc0\u0dcf \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db8\u0dd4. \u0d9a\u0dbd\u0dd2\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0db1\u0dca \u0d9a\u0dc5 \u0db4\u0dbb\u0dd2\u0daf\u0dd2, \u0da7\u0ddd\u0d9a\u0db1\u0dd3\u0d9a\u0dbb\u0dab\u0dba\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d85\u0db4\u0dd2 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0d9a\u0da9\u0dcf \u0daf\u0dd0\u0db8\u0dd4\u0dc0\u0dc4\u0ddc\u0dad\u0dca \u0db8\u0dd9\u0dba \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0db1\u0ddc\u0dc0\u0dda. </p>\n",
|
||||
"<p>Break words </p>\n": "<p>\u0dc0\u0da0\u0db1\u0d9a\u0da9\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate the average token embeddings. Note that the attention mask is <span translate=no>_^_0_^_</span> if the token is empty padded. We get empty tokens because the chunks are of different lengths. </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0da7\u0ddd\u0d9a\u0db1 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1. \u0da7\u0ddd\u0d9a\u0db1\u0dba \u0dc4\u0dd2\u0dc3\u0dca \u0db4\u0dd1\u0da9\u0dca <span translate=no>_^_0_^_</span> \u0db1\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d86\u0dc0\u0dbb\u0dab \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1. \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc0\u0dd2\u0dc0\u0dd2\u0db0 \u0daf\u0dd2\u0d9c \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d85\u0db4\u0da7 \u0dc4\u0dd2\u0dc3\u0dca \u0da7\u0ddd\u0d9a\u0db1 \u0dbd\u0dd0\u0db6\u0dda. </p>\n",
|
||||
"<p>Check BERT model outputs </p>\n": "<p>\u0db6\u0dbb\u0dca\u0da7\u0dca\u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba\u0db1\u0dca \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Check BERT tokenizer </p>\n": "<p>\u0db6\u0dbb\u0dca\u0da7\u0dca\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Check recreating text from token ids </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dcaid \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0db4\u0dd9\u0dc5 \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Evaluate the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dad\u0d9a\u0dca\u0dc3\u0dda\u0dbb\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get chunk embeddings </p>\n": "<p>\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the token embeddings </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>If empty return original string </p>\n": "<p>\u0dc4\u0dd2\u0dc3\u0dca\u0d86\u0db4\u0dc3\u0dd4 \u0db8\u0dd4\u0dbd\u0dca string \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>Initialize </p>\n": "<p>\u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load the BERT model from <a href=\"https://huggingface.co/bert-base-uncased\">HuggingFace</a> </p>\n": "<p><a href=\"https://huggingface.co/bert-base-uncased\">HuggingFace</a> \u0dc0\u0dd9\u0dad\u0dd2\u0db1\u0dca \u0db6\u0dbb\u0dca\u0da7\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db4\u0da7\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load the BERT tokenizer from <a href=\"https://huggingface.co/bert-base-uncased\">HuggingFace</a> </p>\n": "<p><a href=\"https://huggingface.co/bert-base-uncased\">\u0dc4\u0d9c\u0dca\u0da2\u0dd2\u0d82 \u0dc6\u0dda\u0dc3\u0dca \u0dc0\u0dd9\u0dad\u0dd2\u0db1\u0dca \u0db6\u0dbb\u0dca\u0da7\u0dca \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca</a> \u0db4\u0da7\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move the model to <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc0\u0dd9\u0dad \u0d9c\u0dd9\u0db1 \u0dba\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Move token ids, attention mask and token types to the device </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0dc4\u0dd0\u0db3\u0dd4\u0db1\u0dd4\u0db8\u0dca\u0db4\u0dad\u0dca, \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0d86\u0dc0\u0dbb\u0dab \u0dc3\u0dc4 \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0dc0\u0dbb\u0dca\u0d9c \u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0da7 \u0d9c\u0dd9\u0db1 \u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Otherwise, return the stripped string </p>\n": "<p>\u0d91\u0dc3\u0dda\u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2\u0db1\u0db8\u0dca, \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db1\u0dd6\u0dbd\u0dca \u0d86\u0db4\u0dc3\u0dd4 \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Remove first and last pieces </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0dc4\u0dcf \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0d9a\u0dd1\u0dbd\u0dd2 \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Remove whitespace </p>\n": "<p>\u0dc0\u0dba\u0dd2\u0da7\u0dca\u0dc3\u0dca\u0db4\u0dda\u0dc3\u0dca\u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Sample </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba </p>\n",
|
||||
"<p>Strip whitespace </p>\n": "<p>\u0dad\u0dd3\u0dbb\u0dd4\u0dc0\u0dba\u0dd2\u0da7\u0dca\u0dc3\u0dca\u0db4\u0dda\u0dc3\u0dca </p>\n",
|
||||
"<p>Tokenize the chunks with BERT tokenizer </p>\n": "<p>\u0db6\u0dbb\u0dca\u0da7\u0dca\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0dc3\u0db8\u0d9f \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0da7\u0ddd\u0d9a\u0db1\u0dca\u0da7\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Trim the chunks </p>\n": "<p>\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0d9a\u0db4\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>We don't need to compute gradients </p>\n": "<p>\u0d85\u0db4\u0da7\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0db1\u0dd0\u0dad </p>\n",
|
||||
"BERT Embeddings of chunks of text": "\u0db4\u0dd9\u0dc5 \u0d9a\u0dd0\u0db6\u0dbd\u0dd2 \u0dc0\u0dbd \u0db6\u0dbb\u0dca\u0da7\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca",
|
||||
"Generate BERT embeddings for chunks using a frozen BERT model": "\u0dc1\u0dd3\u0dad \u0d9a\u0dc5 BERT \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd2\u0db1\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc3\u0db3\u0dc4\u0dcf BERT \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0da2\u0db1\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"<h1>BERT Embeddings of chunks of text</h1>\n<p>This is the code to get BERT embeddings of chunks for <a href=\"index.html\">RETRO model</a>.</p>\n": "<h1>BERT \u6587\u672c\u5757\u7684\u5d4c\u5165</h1>\n<p>\u8fd9\u662f\u83b7\u53d6 <a href=\"index.html\">RETRO \u6a21\u578b</a>\u5757\u7684 BERT \u5d4c\u5165\u7684\u4ee3\u7801\u3002</p>\n",
|
||||
"<h2>BERT Embeddings</h2>\n<p>For a given chunk of text <span translate=no>_^_0_^_</span> this class generates BERT embeddings <span translate=no>_^_1_^_</span>. <span translate=no>_^_2_^_</span> is the average of BERT embeddings of all the tokens in <span translate=no>_^_3_^_</span>.</p>\n": "<h2>BERT \u5d4c\u5165\u5f0f</h2>\n<p>\u5bf9\u4e8e\u7ed9\u5b9a\u7684\u6587\u672c\u5757\uff0c<span translate=no>_^_0_^_</span>\u8fd9\u4e2a\u7c7b\u4f1a\u751f\u6210 BERT \u5d4c\u5165<span translate=no>_^_1_^_</span>\u3002<span translate=no>_^_2_^_</span>\u662f\u6240\u6709\u4ee4\u724c\u7684 BERT \u5d4c\u5165\u7684\u5e73\u5747\u503c<span translate=no>_^_3_^_</span>\u3002</p>\n",
|
||||
"<h3>Code to test BERT embeddings</h3>\n": "<h3>\u7528\u4e8e\u6d4b\u8bd5 BERT \u5d4c\u5165\u7684\u4ee3\u7801</h3>\n",
|
||||
"<h3>Get <span translate=no>_^_0_^_</span> for a list of chunks.</h3>\n": "<h3><span translate=no>_^_0_^_</span>\u83b7\u53d6\u533a\u5757\u5217\u8868\u3002</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> In this implementation, we do not make chunks with a fixed number of tokens. One of the reasons is that this implementation uses character-level tokens and BERT uses its sub-word tokenizer.</p>\n<p>So this method will truncate the text to make sure there are no partial tokens.</p>\n<p>For instance, a chunk could be like <span translate=no>_^_0_^_</span>, with partial words (partial sub-word tokens) on the ends. We strip them off to get better BERT embeddings. As mentioned earlier this is not necessary if we broke chunks after tokenizing.</p>\n": "<p>\u5728\u6b64\u5b9e\u73b0\u4e2d\uff0c\u6211\u4eec\u4e0d\u4f1a\u4f7f\u7528\u56fa\u5b9a\u6570\u91cf\u7684\u4ee4\u724c\u5236\u4f5c\u533a\u5757\u3002\u539f\u56e0\u4e4b\u4e00\u662f\u6b64\u5b9e\u73b0\u4f7f\u7528\u5b57\u7b26\u7ea7\u4ee4\u724c\uff0c\u800c BERT \u4f7f\u7528\u5176\u5b50\u8bcd\u5206\u8bcd\u5668\u3002</p>\n<p>\u56e0\u6b64\uff0c\u6b64\u65b9\u6cd5\u5c06\u622a\u65ad\u6587\u672c\u4ee5\u786e\u4fdd\u6ca1\u6709\u90e8\u5206\u6807\u8bb0\u3002</p>\n<p>\u4f8b\u5982\uff0c\u4e00\u4e2a\u5757\u53ef\u80fd\u50cf<span translate=no>_^_0_^_</span>\uff0c\u672b\u5c3e\u5e26\u6709\u90e8\u5206\u5355\u8bcd\uff08\u90e8\u5206\u5b50\u8bcd\u6807\u8bb0\uff09\u3002\u6211\u4eec\u5265\u79bb\u5b83\u4eec\u4ee5\u83b7\u5f97\u66f4\u597d\u7684 BERT \u5d4c\u5165\u3002\u5982\u524d\u6240\u8ff0\uff0c\u5982\u679c\u6211\u4eec\u5728\u6807\u8bb0\u5316\u540e\u7834\u574f\u4e86\u533a\u5757\uff0c\u5219\u6ca1\u6709\u5fc5\u8981\u8fd9\u6837\u505a\u3002</p>\n",
|
||||
"<p>Break words </p>\n": "<p>\u65ad\u8bcd</p>\n",
|
||||
"<p>Calculate the average token embeddings. Note that the attention mask is <span translate=no>_^_0_^_</span> if the token is empty padded. We get empty tokens because the chunks are of different lengths. </p>\n": "<p>\u8ba1\u7b97\u5e73\u5747\u4ee3\u5e01\u5d4c\u5165\u91cf\u3002\u8bf7\u6ce8\u610f\uff0c<span translate=no>_^_0_^_</span>\u5982\u679c\u4ee4\u724c\u662f\u7a7a\u586b\u5145\u7684\uff0c\u5219\u6ce8\u610f\u63a9\u7801\u4e3a\u3002\u6211\u4eec\u5f97\u5230\u7a7a\u4ee4\u724c\uff0c\u56e0\u4e3a\u8fd9\u4e9b\u5757\u7684\u957f\u5ea6\u4e0d\u540c\u3002</p>\n",
|
||||
"<p>Check BERT model outputs </p>\n": "<p>\u68c0\u67e5 BERT \u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>Check BERT tokenizer </p>\n": "<p>\u67e5\u770b BERT \u5206\u8bcd\u5668</p>\n",
|
||||
"<p>Check recreating text from token ids </p>\n": "<p>\u68c0\u67e5\u4ece\u4ee4\u724c ID \u4e2d\u91cd\u65b0\u521b\u5efa\u6587\u672c</p>\n",
|
||||
"<p>Evaluate the model </p>\n": "<p>\u8bc4\u4f30\u6a21\u578b</p>\n",
|
||||
"<p>Get chunk embeddings </p>\n": "<p>\u83b7\u53d6\u533a\u5757\u5d4c\u5165</p>\n",
|
||||
"<p>Get the token embeddings </p>\n": "<p>\u83b7\u53d6\u4ee4\u724c\u5d4c\u5165</p>\n",
|
||||
"<p>If empty return original string </p>\n": "<p>\u5982\u679c\u4e3a\u7a7a\u5219\u8fd4\u56de\u539f\u59cb\u5b57\u7b26\u4e32</p>\n",
|
||||
"<p>Initialize </p>\n": "<p>\u521d\u59cb\u5316</p>\n",
|
||||
"<p>Load the BERT model from <a href=\"https://huggingface.co/bert-base-uncased\">HuggingFace</a> </p>\n": "<p>\u4ece <a href=\"https://huggingface.co/bert-base-uncased\">HuggingFac</a> e \u52a0\u8f7d BERT \u6a21\u578b</p>\n",
|
||||
"<p>Load the BERT tokenizer from <a href=\"https://huggingface.co/bert-base-uncased\">HuggingFace</a> </p>\n": "<p>\u4ece <a href=\"https://huggingface.co/bert-base-uncased\">HuggingFac</a> e \u52a0\u8f7d BERT \u5206\u8bcd\u5668</p>\n",
|
||||
"<p>Move the model to <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5c06\u6a21\u578b\u79fb\u5230<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Move token ids, attention mask and token types to the device </p>\n": "<p>\u5c06\u4ee4\u724c ID\u3001\u6ce8\u610f\u63a9\u7801\u548c\u4ee4\u724c\u7c7b\u578b\u79fb\u52a8\u5230\u8bbe\u5907</p>\n",
|
||||
"<p>Otherwise, return the stripped string </p>\n": "<p>\u5426\u5219\uff0c\u8fd4\u56de\u88ab\u5265\u79bb\u7684\u5b57\u7b26\u4e32</p>\n",
|
||||
"<p>Remove first and last pieces </p>\n": "<p>\u79fb\u9664\u7b2c\u4e00\u5757\u548c\u6700\u540e\u4e00\u5757\u788e\u7247</p>\n",
|
||||
"<p>Remove whitespace </p>\n": "<p>\u79fb\u9664\u7a7a\u683c</p>\n",
|
||||
"<p>Sample </p>\n": "<p>\u6837\u672c</p>\n",
|
||||
"<p>Strip whitespace </p>\n": "<p>\u53bb\u6389\u7a7a\u767d</p>\n",
|
||||
"<p>Tokenize the chunks with BERT tokenizer </p>\n": "<p>\u4f7f\u7528 BERT \u5206\u8bcd\u5668\u5bf9\u533a\u5757\u8fdb\u884c\u6807\u8bb0\u5316</p>\n",
|
||||
"<p>Trim the chunks </p>\n": "<p>\u4fee\u526a\u5757</p>\n",
|
||||
"<p>We don't need to compute gradients </p>\n": "<p>\u6211\u4eec\u4e0d\u9700\u8981\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"BERT Embeddings of chunks of text": "BERT \u6587\u672c\u5757\u7684\u5d4c\u5165",
|
||||
"Generate BERT embeddings for chunks using a frozen BERT model": "\u4f7f\u7528\u51bb\u7ed3 BERT \u6a21\u578b\u4e3a\u533a\u5757\u751f\u6210 BERT \u5d4c\u5165"
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"<h1>Database for nearest neighbor retrieval</h1>\n<p>This is the build the database and retrieves nearest neighbors for <a href=\"index.html\">RETRO model</a>.</p>\n<p>We use <a href=\"https://faiss.ai/\">FAISS library</a> for the database whilst the paper had used the SCaNN library.</p>\n": "<h1>\u6700\u8fd1\u508d\u691c\u7d22\u7528\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9</h1>\n<p>\u3053\u308c\u306f\u3001<a href=\"index.html\">RETRO\u30e2\u30c7\u30eb\u306e\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u3092\u69cb\u7bc9\u3057</a>\u3001\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u3092\u691c\u7d22\u3059\u308b\u3082\u306e\u3067\u3059\u3002</p>\n</a><p>\u8ad6\u6587\u3067\u306fScanN\u30e9\u30a4\u30d6\u30e9\u30ea\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3057\u305f\u304c\u3001<a href=\"https://faiss.ai/\">\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u306b\u306fFAISS\u30e9\u30a4\u30d6\u30e9\u30ea\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<h2>Build Database</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the length of a chunk (number of characters) </li>\n<li><span translate=no>_^_1_^_</span> is the batch size to use when calculating <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the number of features in <span translate=no>_^_4_^_</span> embeddings <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">lists to select in FAISS index</a> </li>\n<li><span translate=no>_^_5_^_</span> is the number of lists in the index </li>\n<li><span translate=no>_^_6_^_</span> encoded vector size in the index </li>\n<li><span translate=no>_^_7_^_</span> is the number of lists to probe </li>\n<li>`n_train' is the number of keys to train the index on</li></ul>\n": "<h2>\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u3092\u69cb\u7bc9</h2>\n<ul><li><span translate=no>_^_0_^_</span>\u30c1\u30e3\u30f3\u30af\u306e\u9577\u3055 (\u6587\u5b57\u6570)</li>\n<li><span translate=no>_^_1_^_</span>\u8a08\u7b97\u6642\u306b\u4f7f\u7528\u3059\u308b\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba\u3067\u3059 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>FAISS <span translate=no>_^_4_^_</span> <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3067\u9078\u629e\u3059\u308b\u57cb\u3081\u8fbc\u307f\u30ea\u30b9\u30c8\u5185\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u6570\u3067\u3059</a></li>\n<li><span translate=no>_^_5_^_</span>\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u5185\u306e\u30ea\u30b9\u30c8\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_6_^_</span>\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u5185\u306e\u30a8\u30f3\u30b3\u30fc\u30c9\u3055\u308c\u305f\u30d9\u30af\u30c8\u30eb\u30b5\u30a4\u30ba</li>\n<li><span translate=no>_^_7_^_</span>\u306f\u8abf\u3079\u308b\u30ea\u30b9\u30c8\u306e\u6570\u3067\u3059</li>\n<li>`n_train' \u306f\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u30ad\u30fc\u306e\u6570\u3067\u3059</li></ul>\n",
|
||||
"<h2>Index for retrieving nearest neighbors</h2>\n": "<h2>\u6700\u8fd1\u508d\u3092\u691c\u7d22\u3059\u308b\u305f\u3081\u306e\u30a4\u30f3\u30c7\u30c3\u30af\u30b9</h2>\n",
|
||||
"<h3>Retrieve nearest neighbors</h3>\n": "<h3>\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u3092\u691c\u7d22\u3059\u308b</h3>\n",
|
||||
"<h4>Filter neighbors that overlap with the query</h4>\n<p>The positions of the neighbors are given by <span translate=no>_^_0_^_</span> and the position of the query chunk is <span translate=no>_^_1_^_</span>.</p>\n": "<h4>\u30af\u30a8\u30ea\u3068\u91cd\u8907\u3059\u308b\u8fd1\u508d\u3092\u30d5\u30a3\u30eb\u30bf\u30ea\u30f3\u30b0\u3059\u308b</h4>\n<p><span translate=no>_^_0_^_</span>\u8fd1\u508d\u306e\u4f4d\u7f6e\u306f\u3067\u6307\u5b9a\u3055\u308c\u3001\u30af\u30a8\u30ea\u30c1\u30e3\u30f3\u30af\u306e\u4f4d\u7f6e\u306f\u3067\u3059\u3002<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Add the chunks to the index in batches of size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u3092\u30b5\u30a4\u30ba\u3054\u3068\u306b\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u306b\u8ffd\u52a0\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Add to index </p>\n": "<p>\u7d22\u5f15\u306b\u8ffd\u52a0</p>\n",
|
||||
"<p>Create the <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">FAISS index</a> </p>\n": "<p><a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">FAISS</a> \u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u4f5c\u6210\u3059\u308b</p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> nearest neighbors from the database </p>\n": "<p><span translate=no>_^_0_^_</span>\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u304b\u3089\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> of query chunks </p>\n": "<p><span translate=no>_^_0_^_</span>\u30af\u30a8\u30ea\u30c1\u30e3\u30f3\u30af\u306e\u53d6\u5f97</p>\n",
|
||||
"<p>Get a random sample of the the chunk indexes </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u306e\u30e9\u30f3\u30c0\u30e0\u30b5\u30f3\u30d7\u30eb\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get chunk embeddings by processing <span translate=no>_^_0_^_</span> number of chunks on each iteration </p>\n": "<p><span translate=no>_^_0_^_</span>\u5404\u53cd\u5fa9\u51e6\u7406\u3067\u30c1\u30e3\u30f3\u30af\u306e\u6570\u3092\u51e6\u7406\u3057\u3066\u30c1\u30e3\u30f3\u30af\u306e\u57cb\u3081\u8fbc\u307f\u3092\u884c\u3046</p>\n",
|
||||
"<p>Get the closest <span translate=no>_^_0_^_</span> after filtering </p>\n": "<p><span translate=no>_^_0_^_</span>\u30d5\u30a3\u30eb\u30bf\u30ea\u30f3\u30b0\u5f8c\u306b\u6700\u3082\u8fd1\u3044\u3082\u306e\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the offsets of each of the chunks </p>\n": "<p>\u5404\u30c1\u30e3\u30f3\u30af\u306e\u30aa\u30d5\u30bb\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get training data (a string) </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30c7\u30fc\u30bf (\u6587\u5b57\u5217) \u3092\u53d6\u5f97</p>\n",
|
||||
"<p>If the query chunk offsets are given filter out overlapping chunks </p>\n": "<p>\u30af\u30a8\u30ea\u306e\u30c1\u30e3\u30f3\u30af\u30aa\u30d5\u30bb\u30c3\u30c8\u304c\u6307\u5b9a\u3055\u308c\u3066\u3044\u308b\u5834\u5408\u306f\u3001\u91cd\u8907\u3059\u308b\u30c1\u30e3\u30f3\u30af\u3092\u9664\u5916\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Initialize BERT to get <span translate=no>_^_0_^_</span> </p>\n": "<p>BERT \u3092\u521d\u671f\u5316\u3057\u3066\u53d6\u5f97 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Load the database </p>\n": "<p>\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u3092\u8aad\u307f\u8fbc\u3080</p>\n",
|
||||
"<p>Load the dataset text file </p>\n": "<p>\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u30c6\u30ad\u30b9\u30c8\u30d5\u30a1\u30a4\u30eb\u3092\u8aad\u307f\u8fbc\u3080</p>\n",
|
||||
"<p>Merge them into a single tensor </p>\n": "<p>\u305d\u308c\u3089\u3092\u5358\u4e00\u306e\u30c6\u30f3\u30bd\u30eb\u306b\u30de\u30fc\u30b8\u3057\u307e\u3059</p>\n",
|
||||
"<p>Number of chunks </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u306e\u6570</p>\n",
|
||||
"<p>Save the index </p>\n": "<p>\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u4fdd\u5b58\u3059\u308b</p>\n",
|
||||
"<p>Split the text into chunks of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30c6\u30ad\u30b9\u30c8\u3092\u6b21\u306e\u30c1\u30e3\u30f3\u30af\u306b\u5206\u5272\u3057\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Train the index to store the keys </p>\n": "<p>\u30ad\u30fc\u3092\u4fdd\u5b58\u3059\u308b\u3088\u3046\u306b\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the chunk length </li>\n<li><span translate=no>_^_1_^_</span> is the number of lists to probe </li>\n<li><span translate=no>_^_2_^_</span> is the number of neighbors to retrieve </li>\n<li><span translate=no>_^_3_^_</span> is the number of extra neighbors to retrieve since we will be removing neighbors overlapping with the query chunk </li>\n<li><span translate=no>_^_4_^_</span> is the extra text length to avoid when checking for overlaps</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u30c1\u30e3\u30f3\u30af\u306e\u9577\u3055\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u8abf\u3079\u308b\u30ea\u30b9\u30c8\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u53d6\u5f97\u3059\u308b\u8fd1\u508d\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_3_^_</span>\u30af\u30a8\u30ea\u30c1\u30e3\u30f3\u30af\u3068\u91cd\u8907\u3057\u3066\u3044\u308b\u8fd1\u508d\u3092\u524a\u9664\u3059\u308b\u3053\u3068\u306b\u306a\u308b\u305f\u3081\u3001\u53d6\u5f97\u3059\u308b\u4f59\u5206\u306a\u8fd1\u508d\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_4_^_</span>\u30aa\u30fc\u30d0\u30fc\u30e9\u30c3\u30d7\u3092\u30c1\u30a7\u30c3\u30af\u3059\u308b\u3068\u304d\u306b\u907f\u3051\u308b\u3079\u304d\u4f59\u5206\u306a\u30c6\u30ad\u30b9\u30c8\u9577\u3067\u3059</li></ul>\n",
|
||||
"Database for nearest neighbor retrieval": "\u6700\u8fd1\u508d\u691c\u7d22\u7528\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9",
|
||||
"Nearest neighbor retrieval and creation of the database": "\u6700\u8fd1\u508d\u691c\u7d22\u3068\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u306e\u4f5c\u6210"
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"<h1>Database for nearest neighbor retrieval</h1>\n<p>This is the build the database and retrieves nearest neighbors for <a href=\"index.html\">RETRO model</a>.</p>\n<p>We use <a href=\"https://faiss.ai/\">FAISS library</a> for the database whilst the paper had used the SCaNN library.</p>\n": "<h1>\u0dc5\u0d9f\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba</h1>\n<p>\u0db8\u0dd9\u0dba\u0dc0\u0dda \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 \u0dc3\u0dc4 <a href=\"index.html\">RETRO \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</a>\u0dc3\u0db3\u0dc4\u0dcf \u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dd3. </p>\n</a> <p>\u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2ScanN \u0db4\u0dd4\u0dc3\u0dca\u0dad\u0d9a\u0dcf\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 <a href=\"https://faiss.ai/\">FaISS \u0db4\u0dd4\u0dc3\u0dca\u0dad\u0d9a\u0dcf\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
|
||||
"<h2>Build Database</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the length of a chunk (number of characters) </li>\n<li><span translate=no>_^_1_^_</span> is the batch size to use when calculating <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the number of features in <span translate=no>_^_4_^_</span> embeddings <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">lists to select in FAISS index</a> </li>\n<li><span translate=no>_^_5_^_</span> is the number of lists in the index </li>\n<li><span translate=no>_^_6_^_</span> encoded vector size in the index </li>\n<li><span translate=no>_^_7_^_</span> is the number of lists to probe </li>\n<li>`n_train' is the number of keys to train the index on</li></ul>\n": "<h2>\u0daf\u0dad\u0dca\u0dad\u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0d9c\u0ddc\u0da9\u0db1\u0dd0\u0d9c\u0dd3\u0db8</h2>\n<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a \u0daf\u0dd2\u0d9c (\u0d85\u0d9a\u0dca\u0dc2\u0dbb \u0d9c\u0dab\u0db1) </li>\n<li><span translate=no>_^_1_^_</span> \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dba\u0dd2 <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">FAISS \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba\u0dda \u0dad\u0ddd\u0dbb\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf <span translate=no>_^_4_^_</span> \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4 \u0dc0\u0dbd</a> \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 \u0dc0\u0dda </li>\n<li><span translate=no>_^_5_^_</span> \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba\u0dda \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4 \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_6_^_</span> \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba\u0dda \u0d9a\u0dda\u0dad\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0daf\u0ddb\u0dc1\u0dd2\u0d9a \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </li>\n<li><span translate=no>_^_7_^_</span> \u0dc0\u0dd2\u0db8\u0dbb\u0dca\u0dc1\u0db1\u0dba \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4 \u0d9c\u0dab\u0db1 </li>\n<li>`n_train'\u0dba\u0db1\u0dd4 \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dba\u0dad\u0dd4\u0dbb\u0dd4 \u0d9c\u0dab\u0db1 \u0dc0\u0dda</li></ul>\n",
|
||||
"<h2>Index for retrieving nearest neighbors</h2>\n": "<h2>\u0dc5\u0d9f\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba</h2>\n",
|
||||
"<h3>Retrieve nearest neighbors</h3>\n": "<h3>\u0dc5\u0d9f\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h4>Filter neighbors that overlap with the query</h4>\n<p>The positions of the neighbors are given by <span translate=no>_^_0_^_</span> and the position of the query chunk is <span translate=no>_^_1_^_</span>.</p>\n": "<h4>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0dc3\u0db8\u0d9f \u0d85\u0dad\u0dd2\u0da0\u0dca\u0da1\u0dcf\u0daf\u0db1\u0dba \u0dc0\u0db1 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0db4\u0dd9\u0dbb\u0dc4\u0db1\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1</h4>\n<p>\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dcf\u0dc3\u0dd3\u0db1\u0dca\u0d9c\u0dda\u0dad\u0db1\u0dad\u0dd4\u0dbb\u0dd4 \u0dbd\u0db6\u0dcf \u0daf\u0dd3 \u0d87\u0dad\u0dd2 <span translate=no>_^_0_^_</span> \u0d85\u0dad\u0dbb \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0dda \u0db4\u0dd2\u0dc4\u0dd2\u0da7\u0dd3\u0db8 \u0dc0\u0dda <span translate=no>_^_1_^_</span>. </p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Add the chunks to the index in batches of size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dd9\u0db1\u0dca\u0d9a\u0dcf\u0dab\u0dca\u0da9\u0dc0\u0dbd \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba\u0da7 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Add to index </p>\n": "<p>\u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba\u0da7\u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create the <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">FAISS index</a> </p>\n": "<p><a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">FAISS \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba</a> \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> nearest neighbors from the database </p>\n": "<p>\u0daf\u0dad\u0dca\u0dad\u0d9c\u0db6\u0da9\u0dcf\u0dc0\u0dd9\u0db1\u0dca <span translate=no>_^_0_^_</span> \u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> of query chunks </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0dca\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get a random sample of the the chunk indexes </p>\n": "<p>\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0db8\u0dca\u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dc0\u0dbd \u0d85\u0dc4\u0db9\u0dd4 \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba\u0d9a\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get chunk embeddings by processing <span translate=no>_^_0_^_</span> number of chunks on each iteration </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0db4\u0dd4\u0db1\u0dbb\u0dcf\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba\u0dda \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 <span translate=no>_^_0_^_</span> \u0d9c\u0dab\u0db1 \u0dc3\u0dd0\u0d9a\u0dc3\u0dd3\u0db8 \u0db8\u0d9c\u0dd2\u0db1\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the closest <span translate=no>_^_0_^_</span> after filtering </p>\n": "<p><span translate=no>_^_0_^_</span> \u0db4\u0dd9\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0d86\u0dc3\u0db1\u0dca\u0db1\u0dad\u0db8 \u0daf\u0dda \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the offsets of each of the chunks </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc0\u0dbd \u0d95\u0dc6\u0dca\u0dc3\u0dd9\u0da7\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get training data (a string) </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0daf\u0dad\u0dca\u0dad \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 (\u0db1\u0dd6\u0dbd\u0d9a\u0dca) </p>\n",
|
||||
"<p>If the query chunk offsets are given filter out overlapping chunks </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0dda \u0d95\u0dc6\u0dca\u0dc3\u0dd9\u0da7\u0dca \u0d85\u0dad\u0dd2\u0da0\u0dca\u0da1\u0dcf\u0daf\u0db1\u0dba \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc3\u0dd2\u0daf\u0dd4 \u0db4\u0dd9\u0dbb\u0dc4\u0db1 \u0dbd\u0db6\u0dcf \u0daf\u0dd3 \u0dad\u0dd2\u0db6\u0dda \u0db1\u0db8\u0dca </p>\n",
|
||||
"<p>Initialize BERT to get <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db6\u0dbb\u0dca\u0da7\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Load the database </p>\n": "<p>\u0daf\u0dad\u0dca\u0dad\u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db4\u0da7\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load the dataset text file </p>\n": "<p>\u0daf\u0dad\u0dca\u0dad\u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db4\u0dd9\u0dc5 \u0d9c\u0ddc\u0db1\u0dd4\u0dc0 \u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Merge them into a single tensor </p>\n": "<p>\u0d92\u0dc0\u0dcf\u0dad\u0db1\u0dd2 \u0d86\u0dad\u0db1\u0dba\u0d9a\u0da7 \u0d92\u0d9a\u0dcf\u0db6\u0daf\u0dca\u0db0 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Number of chunks </p>\n": "<p>\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Save the index </p>\n": "<p>\u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba\u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Split the text into chunks of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dd9\u0dc5\u0d9a\u0dd0\u0db6\u0dbd\u0dd2 \u0dc0\u0dbd\u0da7 \u0db6\u0dd9\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Train the index to store the keys </p>\n": "<p>\u0dba\u0dad\u0dd4\u0dbb\u0dd4\u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the chunk length </li>\n<li><span translate=no>_^_1_^_</span> is the number of lists to probe </li>\n<li><span translate=no>_^_2_^_</span> is the number of neighbors to retrieve </li>\n<li><span translate=no>_^_3_^_</span> is the number of extra neighbors to retrieve since we will be removing neighbors overlapping with the query chunk </li>\n<li><span translate=no>_^_4_^_</span> is the extra text length to avoid when checking for overlaps</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0d9a\u0dd4\u0da7\u0dd2\u0dba\u0dda \u0daf\u0dd2\u0d9c \u0dc0\u0dda </li>\n<li><span translate=no>_^_1_^_</span> \u0dc0\u0dd2\u0db8\u0dbb\u0dca\u0dc1\u0db1\u0dba \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4 \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4 \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_2_^_</span> \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0 </li>\n<li><span translate=no>_^_3_^_</span> \u0d85\u0db4\u0dd2 \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0dda \u0dc3\u0db8\u0d9c \u0d85\u0dad\u0dd2\u0da0\u0dca\u0da1\u0dcf\u0daf\u0db1\u0dba \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad \u0dc3\u0dd2\u0da7 \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0d85\u0db8\u0dad\u0dbb \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0 \u0dc0\u0dda </li>\n<li><span translate=no>_^_4_^_</span> \u0d85\u0dad\u0dd2\u0da0\u0dca\u0da1\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3 \u0dc0\u0dc5\u0d9a\u0dca\u0dc0\u0dcf \u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4 \u0d85\u0db8\u0dad\u0dbb \u0db4\u0dd9\u0dc5 \u0daf\u0dd2\u0d9c \u0dc0\u0dda</li></ul>\n",
|
||||
"Database for nearest neighbor retrieval": "\u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba",
|
||||
"Nearest neighbor retrieval and creation of the database": "\u0d86\u0dc3\u0db1\u0dca\u0db1\u0dad\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0dc4 \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8"
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"<h1>Database for nearest neighbor retrieval</h1>\n<p>This is the build the database and retrieves nearest neighbors for <a href=\"index.html\">RETRO model</a>.</p>\n<p>We use <a href=\"https://faiss.ai/\">FAISS library</a> for the database whilst the paper had used the SCaNN library.</p>\n": "<h1>\u6700\u8fd1\u90bb\u68c0\u7d22\u7684\u6570\u636e\u5e93</h1>\n<p>\u8fd9\u662f\u6784\u5efa\u6570\u636e\u5e93\u5e76\u68c0\u7d22 RETRO <a href=\"index.html\">\u6a21\u578b\u7684\u6700\u8fd1\u90bb\u57df</a>\u3002</p>\n</a><p>\u6211\u4eec\u4f7f\u7528 <a href=\"https://faiss.ai/\">FAISS \u5e93\u4f5c\u4e3a\u6570\u636e\u5e93\uff0c\u800c\u8bba\u6587\u4f7f\u7528\u4e86 ScanN \u5e93\u3002</p>\n",
|
||||
"<h2>Build Database</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the length of a chunk (number of characters) </li>\n<li><span translate=no>_^_1_^_</span> is the batch size to use when calculating <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> is the number of features in <span translate=no>_^_4_^_</span> embeddings <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">lists to select in FAISS index</a> </li>\n<li><span translate=no>_^_5_^_</span> is the number of lists in the index </li>\n<li><span translate=no>_^_6_^_</span> encoded vector size in the index </li>\n<li><span translate=no>_^_7_^_</span> is the number of lists to probe </li>\n<li>`n_train' is the number of keys to train the index on</li></ul>\n": "<h2>\u5efa\u7acb\u6570\u636e\u5e93</h2>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u5757\u7684\u957f\u5ea6\uff08\u5b57\u7b26\u6570\uff09</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u8ba1\u7b97\u65f6\u8981\u4f7f\u7528\u7684\u6279\u6b21\u5927\u5c0f<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f\u8981\u5728 <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">FAISS \u7d22\u5f15\u4e2d\u9009\u62e9\u7684<span translate=no>_^_4_^_</span>\u5d4c\u5165\u5217\u8868\u4e2d\u8981</a>\u7d20\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_5_^_</span>\u662f\u7d22\u5f15\u4e2d\u7684\u5217\u8868\u6570</li>\n<li><span translate=no>_^_6_^_</span>\u7d22\u5f15\u4e2d\u7684\u7f16\u7801\u5411\u91cf\u5927\u5c0f</li>\n<li><span translate=no>_^_7_^_</span>\u662f\u8981\u63a2\u6d4b\u7684\u5217\u8868\u7684\u6570\u91cf</li>\n<li>`n_train'\u662f\u7528\u4e8e\u8bad\u7ec3\u7d22\u5f15\u7684\u952e\u7684\u6570\u91cf</li></ul>\n",
|
||||
"<h2>Index for retrieving nearest neighbors</h2>\n": "<h2>\u68c0\u7d22\u6700\u8fd1\u90bb\u5c45\u7684\u7d22\u5f15</h2>\n",
|
||||
"<h3>Retrieve nearest neighbors</h3>\n": "<h3>\u68c0\u7d22\u6700\u8fd1\u7684\u90bb\u5c45</h3>\n",
|
||||
"<h4>Filter neighbors that overlap with the query</h4>\n<p>The positions of the neighbors are given by <span translate=no>_^_0_^_</span> and the position of the query chunk is <span translate=no>_^_1_^_</span>.</p>\n": "<h4>\u7b5b\u9009\u4e0e\u67e5\u8be2\u91cd\u53e0\u7684\u90bb\u57df</h4>\n<p>\u90bb\u5c45\u7684\u4f4d\u7f6e\u7531\u7ed9\u51fa<span translate=no>_^_0_^_</span>\uff0c\u67e5\u8be2\u5757\u7684\u4f4d\u7f6e\u4e3a<span translate=no>_^_1_^_</span>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Add the chunks to the index in batches of size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5c06\u5757\u6309\u5927\u5c0f\u6279\u6b21\u6dfb\u52a0\u5230\u7d22\u5f15\u4e2d<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Add to index </p>\n": "<p>\u6dfb\u52a0\u5230\u7d22\u5f15</p>\n",
|
||||
"<p>Create the <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">FAISS index</a> </p>\n": "<p>\u521b\u5efa <a href=\"https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\">FAISS \u6307\u6570</a></p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> nearest neighbors from the database </p>\n": "<p>\u4ece\u6570\u636e\u5e93\u4e2d\u83b7\u53d6<span translate=no>_^_0_^_</span>\u6700\u8fd1\u7684\u90bb\u5c45</p>\n",
|
||||
"<p>Get <span translate=no>_^_0_^_</span> of query chunks </p>\n": "<p>\u83b7\u53d6<span translate=no>_^_0_^_</span>\u67e5\u8be2\u533a\u5757</p>\n",
|
||||
"<p>Get a random sample of the the chunk indexes </p>\n": "<p>\u83b7\u53d6\u533a\u5757\u7d22\u5f15\u7684\u968f\u673a\u6837\u672c</p>\n",
|
||||
"<p>Get chunk embeddings by processing <span translate=no>_^_0_^_</span> number of chunks on each iteration </p>\n": "<p>\u901a\u8fc7\u5904\u7406\u6bcf\u6b21\u8fed\u4ee3\u7684\u5757<span translate=no>_^_0_^_</span>\u6570\u6765\u83b7\u53d6\u533a\u5757\u5d4c\u5165</p>\n",
|
||||
"<p>Get the closest <span translate=no>_^_0_^_</span> after filtering </p>\n": "<p>\u7b5b\u9009<span translate=no>_^_0_^_</span>\u540e\u83b7\u53d6\u6700\u63a5\u8fd1\u7684\u503c</p>\n",
|
||||
"<p>Get the offsets of each of the chunks </p>\n": "<p>\u83b7\u53d6\u6bcf\u4e2a\u533a\u5757\u7684\u504f\u79fb\u91cf</p>\n",
|
||||
"<p>Get training data (a string) </p>\n": "<p>\u83b7\u53d6\u8bad\u7ec3\u6570\u636e\uff08\u5b57\u7b26\u4e32\uff09</p>\n",
|
||||
"<p>If the query chunk offsets are given filter out overlapping chunks </p>\n": "<p>\u5982\u679c\u7ed9\u51fa\u4e86\u67e5\u8be2\u533a\u5757\u504f\u79fb\u91cf\uff0c\u8bf7\u8fc7\u6ee4\u6389\u91cd\u53e0\u7684\u5757</p>\n",
|
||||
"<p>Initialize BERT to get <span translate=no>_^_0_^_</span> </p>\n": "<p>\u521d\u59cb\u5316 BERT \u4ee5\u83b7\u53d6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Load the database </p>\n": "<p>\u88c5\u8f7d\u6570\u636e\u5e93</p>\n",
|
||||
"<p>Load the dataset text file </p>\n": "<p>\u52a0\u8f7d\u6570\u636e\u96c6\u6587\u672c\u6587\u4ef6</p>\n",
|
||||
"<p>Merge them into a single tensor </p>\n": "<p>\u5c06\u5b83\u4eec\u5408\u5e76\u6210\u5355\u4e2a\u5f20\u91cf</p>\n",
|
||||
"<p>Number of chunks </p>\n": "<p>\u533a\u5757\u6570</p>\n",
|
||||
"<p>Save the index </p>\n": "<p>\u4fdd\u5b58\u7d22\u5f15</p>\n",
|
||||
"<p>Split the text into chunks of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5c06\u6587\u672c\u5206\u6210\u5927\u5757<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Train the index to store the keys </p>\n": "<p>\u8bad\u7ec3\u7d22\u5f15\u4ee5\u5b58\u50a8\u5bc6\u94a5</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the chunk length </li>\n<li><span translate=no>_^_1_^_</span> is the number of lists to probe </li>\n<li><span translate=no>_^_2_^_</span> is the number of neighbors to retrieve </li>\n<li><span translate=no>_^_3_^_</span> is the number of extra neighbors to retrieve since we will be removing neighbors overlapping with the query chunk </li>\n<li><span translate=no>_^_4_^_</span> is the extra text length to avoid when checking for overlaps</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u533a\u5757\u957f\u5ea6</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u8981\u63a2\u6d4b\u7684\u5217\u8868\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u8981\u68c0\u7d22\u7684\u90bb\u5c45\u6570\u91cf</li>\n<li><span translate=no>_^_3_^_</span>\u662f\u8981\u68c0\u7d22\u7684\u989d\u5916\u90bb\u5c45\u7684\u6570\u91cf\uff0c\u56e0\u4e3a\u6211\u4eec\u5c06\u79fb\u9664\u4e0e\u67e5\u8be2\u5757\u91cd\u53e0\u7684\u90bb\u5c45</li>\n<li><span translate=no>_^_4_^_</span>\u662f\u68c0\u67e5\u91cd\u53e0\u65f6\u8981\u907f\u514d\u7684\u989d\u5916\u6587\u672c\u957f\u5ea6</li></ul>\n",
|
||||
"Database for nearest neighbor retrieval": "\u6700\u8fd1\u90bb\u68c0\u7d22\u7684\u6570\u636e\u5e93",
|
||||
"Nearest neighbor retrieval and creation of the database": "\u6700\u8fd1\u90bb\u68c0\u7d22\u548c\u521b\u5efa\u6570\u636e\u5e93"
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"<h1>RETRO training dataset</h1>\n<p>We pre-retrieve nearest neighbors from the <a href=\"database.html\">key-value database</a> and create the dataset to train the <a href=\"index.html\">RETRO</a> <a href=\"model.html\">model</a>.</p>\n": "<h1>RETRO \u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8</h1>\n<p><a href=\"database.html\">\u30ad\u30fc\u30d0\u30ea\u30e5\u30fc\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u304b\u3089\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u3092\u4e8b\u524d\u306b\u53d6\u5f97\u3057</a><a href=\"index.html\"><a href=\"model.html\">\u3001RETRO \u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f5c\u6210\u3057\u307e\u3059\u3002</a></a></p>\n",
|
||||
"<h2>Build the dataset</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the chunk length </li>\n<li><span translate=no>_^_1_^_</span> is the number of chunks per training sample </li>\n<li><span translate=no>_^_2_^_</span> is the maximum number of characters to skip between two samples. We skip a few characters between samples to make sure the samples aren't aligned perfectly with the chunks in the <a href=\"database.html\">database</a></li></ul>\n": "<h2>\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u69cb\u7bc9</h2>\n<ul><li><span translate=no>_^_0_^_</span>\u306f\u30c1\u30e3\u30f3\u30af\u306e\u9577\u3055\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b5\u30f3\u30d7\u30eb\u3042\u305f\u308a\u306e\u30c1\u30e3\u30f3\u30af\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>2 \u3064\u306e\u30b5\u30f3\u30d7\u30eb\u9593\u3067\u30b9\u30ad\u30c3\u30d7\u3059\u308b\u6700\u5927\u6587\u5b57\u6570\u3067\u3059\u3002</li></ul><a href=\"database.html\">\u30b5\u30f3\u30d7\u30eb\u304c\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u5185\u306e\u30c1\u30e3\u30f3\u30af\u3068\u5b8c\u5168\u306b\u4e00\u81f4\u3057\u3066\u3044\u306a\u3044\u3053\u3068\u3092\u78ba\u8a8d\u3059\u308b\u305f\u3081\u306b\u3001\u30b5\u30f3\u30d7\u30eb\u9593\u3092\u6570\u6587\u5b57\u30b9\u30ad\u30c3\u30d7\u3057\u3066\u3044\u307e\u3059\u3002</a>\n",
|
||||
"<h2>Dataset</h2>\n<p>This is the PyTorch dataset that loads the dataset created by <span translate=no>_^_0_^_</span>.</p>\n": "<h2>\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8</h2>\n<p>\u3053\u308c\u306f\u3001\u306b\u3088\u3063\u3066\u4f5c\u6210\u3055\u308c\u305f\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u30ed\u30fc\u30c9\u3059\u308b PyTorch \u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3067\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Get a sample</p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u3092\u5165\u624b</p>\n",
|
||||
"<p> Number of samples</p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u6570</p>\n",
|
||||
"<p>Add to list of samples </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u30ea\u30b9\u30c8\u306b\u8ffd\u52a0</p>\n",
|
||||
"<p>Break it into chunks </p>\n": "<p>\u305d\u308c\u3092\u30c1\u30e3\u30f3\u30af\u306b\u5206\u5272\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Collect the offset </p>\n": "<p>\u30aa\u30d5\u30bb\u30c3\u30c8\u306e\u53ce\u96c6</p>\n",
|
||||
"<p>Cursor for the text </p>\n": "<p>\u30c6\u30ad\u30b9\u30c8\u7528\u306e\u30ab\u30fc\u30bd\u30eb</p>\n",
|
||||
"<p>For samples </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u7528</p>\n",
|
||||
"<p>Get neighbor texts. The neighbor length is twice the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8fd1\u6240\u306e\u4eba\u306e\u30c6\u30ad\u30b9\u30c8\u3092\u53d6\u5f97.\u8fd1\u508d\u306e\u9577\u3055\u306f 2 \u500d\u3067\u3059</p>\u3002<span translate=no>_^_0_^_</span>\n",
|
||||
"<p>Get the sample </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u3092\u5165\u624b</p>\n",
|
||||
"<p>Get the sample including an extra character (for prediction) </p>\n": "<p>\u8ffd\u52a0\u6587\u5b57\u3092\u542b\u3080\u30b5\u30f3\u30d7\u30eb\u3092\u53d6\u5f97 (\u4e88\u6e2c\u7528)</p>\n",
|
||||
"<p>Increment the cursor </p>\n": "<p>\u30ab\u30fc\u30bd\u30eb\u3092\u30a4\u30f3\u30af\u30ea\u30e1\u30f3\u30c8\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Iterate through sample offsets </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u30aa\u30d5\u30bb\u30c3\u30c8\u3092\u53cd\u5fa9\u51e6\u7406</p>\n",
|
||||
"<p>Load the index for retrieving neighbors </p>\n": "<p>\u8fd1\u508d\u691c\u7d22\u7528\u306e\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u8aad\u307f\u8fbc\u3080</p>\n",
|
||||
"<p>Load the samples </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u3092\u8aad\u307f\u8fbc\u3080</p>\n",
|
||||
"<p>Load the text file </p>\n": "<p>\u30c6\u30ad\u30b9\u30c8\u30d5\u30a1\u30a4\u30eb\u3092\u8aad\u307f\u8fbc\u3080</p>\n",
|
||||
"<p>Retrieve nearest neighbors </p>\n": "<p>\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u3092\u691c\u7d22\u3059\u308b</p>\n",
|
||||
"<p>Save the samples in JSON. We don't need to use complex dataset storage mechanisms or pre-tokenize since our dataset is small. </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u3092 JSON \u3067\u4fdd\u5b58\u3057\u307e\u3059\u3002\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306f\u5c0f\u3055\u3044\u305f\u3081\u3001\u8907\u96d1\u306a\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u4fdd\u5b58\u30e1\u30ab\u30cb\u30ba\u30e0\u3092\u4f7f\u7528\u3057\u305f\u308a\u3001\u4e8b\u524d\u306b\u30c8\u30fc\u30af\u30f3\u5316\u3057\u305f\u308a\u3059\u308b\u5fc5\u8981\u306f\u3042\u308a\u307e\u305b\u3093</p>\u3002\n",
|
||||
"<p>Skip a few characters to make sure it's not aligned with the neighbors </p>\n": "<p>\u6570\u6587\u5b57\u98db\u3070\u3057\u3066\u3001\u96a3\u306e\u6587\u5b57\u3068\u63c3\u308f\u306a\u3044\u3088\u3046\u306b\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Stop if we've reached the end of the text </p>\n": "<p>\u30c6\u30ad\u30b9\u30c8\u306e\u7d42\u308f\u308a\u306b\u9054\u3057\u305f\u3089\u6b62\u3081\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>The chunk offsets </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u30aa\u30d5\u30bb\u30c3\u30c8</p>\n",
|
||||
"<p>The input </p>\n": "<p>\u30a4\u30f3\u30d7\u30c3\u30c8</p>\n",
|
||||
"<p>The input sample offsets </p>\n": "<p>\u5165\u529b\u30b5\u30f3\u30d7\u30eb\u30aa\u30d5\u30bb\u30c3\u30c8</p>\n",
|
||||
"<p>Tokenize </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u5316</p>\n",
|
||||
"<p>Training portion of it </p>\n": "<p>\u305d\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u90e8\u5206</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the path of the saved JSON file </li>\n<li><span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u4fdd\u5b58\u3055\u308c\u305f JSON \u30d5\u30a1\u30a4\u30eb\u306e\u30d1\u30b9\u3067\u3059</li>\n</ul><li><span translate=no>_^_1_^_</span>\u306f <span translate=no>_^_2_^_</span></li>\n",
|
||||
"Create a dataset for RETRO model training": "RETRO \u30e2\u30c7\u30eb\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7528\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u4f5c\u6210",
|
||||
"Training dataset for RETRO": "RETRO \u7528\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8"
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"<h1>RETRO training dataset</h1>\n<p>We pre-retrieve nearest neighbors from the <a href=\"database.html\">key-value database</a> and create the dataset to train the <a href=\"index.html\">RETRO</a> <a href=\"model.html\">model</a>.</p>\n": "<h1>\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba</h1>\n<p>\u0d85\u0db4\u0dd2\u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1 <a href=\"database.html\">\u0dc0\u0da7\u0dd2\u0db1\u0dcf\u0d9a\u0db8\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9c\u0db6\u0da9\u0dcf\u0dc0\u0dd9\u0db1\u0dca</a> \u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0dbd\u0db6\u0dcf \u0d9c\u0dd9\u0db1 <a href=\"index.html\">RETRO</a> <a href=\"model.html\">\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</a> \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
|
||||
"<h2>Build the dataset</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the chunk length </li>\n<li><span translate=no>_^_1_^_</span> is the number of chunks per training sample </li>\n<li><span translate=no>_^_2_^_</span> is the maximum number of characters to skip between two samples. We skip a few characters between samples to make sure the samples aren't aligned perfectly with the chunks in the <a href=\"database.html\">database</a></li></ul>\n": "<h2>\u0daf\u0dad\u0dca\u0dad\u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h2>\n<ul><li><span translate=no>_^_0_^_</span> \u0d9a\u0dd4\u0da7\u0dd2\u0dba\u0dda \u0daf\u0dd2\u0d9c \u0dc0\u0dda </li>\n<li><span translate=no>_^_1_^_</span> \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba\u0d9a\u0da7 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_2_^_</span> \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0daf\u0dd9\u0d9a\u0d9a\u0dca \u0d85\u0dad\u0dbb \u0db8\u0d9f \u0dc4\u0dd0\u0dbb\u0dd3\u0db8\u0da7 \u0d8b\u0db4\u0dbb\u0dd2\u0db8 \u0d85\u0d9a\u0dca\u0dc2\u0dbb \u0d9c\u0dab\u0db1 \u0dc0\u0dda. \u0d85\u0db4\u0dd2 \u0db8\u0d9f \u0d9a\u0dd2\u0dc4\u0dd2\u0db4\u0dba\u0d9a\u0dca \u0da0\u0dbb\u0dd2\u0dad \u0d85\u0dad\u0dbb \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0db6\u0dc0\u0da7 \u0dc0\u0d9c \u0db6\u0dbd\u0dcf \u0d9c\u0db1\u0dca\u0db1 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0db4\u0dd9\u0dbd\u0d9c\u0dd0\u0dc3\u0dd3 \u0db1\u0ddc\u0db8\u0dd0\u0dad \u0dc3\u0db8\u0d9c \u0dc4\u0ddc\u0db3\u0dd2\u0db1\u0dca \u0d9a\u0dd0\u0db6\u0dbd\u0dd2 \u0daf\u0dd3 <a href=\"database.html\">\u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba</a></li></ul>\n",
|
||||
"<h2>Dataset</h2>\n<p>This is the PyTorch dataset that loads the dataset created by <span translate=no>_^_0_^_</span>.</p>\n": "<h2>\u0daf\u0dad\u0dca\u0dad\u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba</h2>\n<p>\u0dc0\u0dd2\u0dc3\u0dd2\u0db1\u0dca\u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db4\u0da7\u0dc0\u0db1 PyTorch \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db8\u0dd9\u0dba\u0dba\u0dd2 <span translate=no>_^_0_^_</span>. </p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> Get a sample</p>\n": "<p> \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba\u0d9a\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1</p>\n",
|
||||
"<p> Number of samples</p>\n": "<p> \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd\u0d9c\u0dab\u0db1</p>\n",
|
||||
"<p>Add to list of samples </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd\u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0da7 \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Break it into chunks </p>\n": "<p>\u0d91\u0dba\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc0\u0dbd\u0da7 \u0d9a\u0da9\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Collect the offset </p>\n": "<p>\u0d95\u0dc6\u0dca\u0dc3\u0dd9\u0da7\u0dca\u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Cursor for the text </p>\n": "<p>\u0db4\u0dd9\u0dc5\u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0dca\u0dc3\u0dbb\u0dba </p>\n",
|
||||
"<p>For samples </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd\u0dc3\u0db3\u0dc4\u0dcf </p>\n",
|
||||
"<p>Get neighbor texts. The neighbor length is twice the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf\u0d9c\u0dda\u0db4\u0dd9\u0dc5 \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1. \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf\u0d9c\u0dda \u0daf\u0dd2\u0d9c \u0db8\u0dd9\u0db1\u0dca \u0daf\u0dd9\u0d9c\u0dd4\u0dab\u0dba\u0d9a\u0dca \u0dc0\u0dda <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get the sample </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the sample including an extra character (for prediction) </p>\n": "<p>\u0d85\u0db8\u0dad\u0dbb\u0da0\u0dbb\u0dd2\u0dad\u0dba\u0d9a\u0dca \u0d87\u0dad\u0dd4\u0dc5\u0dd4\u0dc0 \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 (\u0d85\u0db1\u0dcf\u0dc0\u0dd0\u0d9a\u0dd2\u0dba \u0dc3\u0db3\u0dc4\u0dcf) </p>\n",
|
||||
"<p>Increment the cursor </p>\n": "<p>\u0d9a\u0dbb\u0dca\u0dc3\u0dbb\u0dba\u0dc0\u0dd0\u0da9\u0dd2 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Iterate through sample offsets </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0d95\u0dc6\u0dca\u0dc3\u0dd9\u0da7\u0dca \u0dc4\u0dbb\u0dc4\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load the index for retrieving neighbors </p>\n": "<p>\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba \u0db4\u0da7\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load the samples </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load the text file </p>\n": "<p>\u0db4\u0dd9\u0dc5\u0d9c\u0ddc\u0db1\u0dd4\u0dc0 \u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Retrieve nearest neighbors </p>\n": "<p>\u0dc5\u0d9f\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Save the samples in JSON. We don't need to use complex dataset storage mechanisms or pre-tokenize since our dataset is small. </p>\n": "<p>JSON\u0dc4\u0dd2 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1. \u0d85\u0db4\u0d9c\u0dda \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0d9a\u0dd4\u0da9\u0dcf \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d85\u0db4\u0da7 \u0dc3\u0d82\u0d9a\u0dd3\u0dbb\u0dca\u0dab \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dba\u0dcf\u0db1\u0dca\u0dad\u0dca\u0dbb\u0dab \u0dc4\u0ddd \u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0da7\u0ddd\u0d9a\u0dd3\u0d9a\u0dbb\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0db1\u0ddc\u0dc0\u0dda. </p>\n",
|
||||
"<p>Skip a few characters to make sure it's not aligned with the neighbors </p>\n": "<p>\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca\u0dc3\u0db8\u0d9f \u0db4\u0dd9\u0dbd\u0d9c\u0dd0\u0dc3\u0dd3 \u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db6\u0dc0\u0da7 \u0dc0\u0d9c \u0db6\u0dbd\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0da0\u0dbb\u0dd2\u0dad \u0d9a\u0dd2\u0dc4\u0dd2\u0db4\u0dba\u0d9a\u0dca \u0db8\u0d9f \u0dc4\u0dbb\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Stop if we've reached the end of the text </p>\n": "<p>\u0d85\u0db4\u0dd2\u0db4\u0dd9\u0dc5\u0dd9\u0dc4\u0dd2 \u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0dba\u0da7 \u0db4\u0dd0\u0db8\u0dd2\u0dab \u0d87\u0dad\u0dca\u0db1\u0db8\u0dca \u0db1\u0dc0\u0dad\u0dca\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>The chunk offsets </p>\n": "<p>\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d95\u0dc6\u0dca\u0dc3\u0dd9\u0da7\u0dca </p>\n",
|
||||
"<p>The input </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0dba </p>\n",
|
||||
"<p>The input sample offsets </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d95\u0dc6\u0dca\u0dc3\u0dd9\u0da7\u0dca </p>\n",
|
||||
"<p>Tokenize </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dca\u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Training portion of it </p>\n": "<p>\u0d91\u0dc4\u0dd2\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0ddc\u0da7\u0dc3 </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the path of the saved JSON file </li>\n<li><span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1 \u0dbd\u0daf JSON \u0d9c\u0ddc\u0db1\u0dd4\u0dc0\u0dda \u0db8\u0dcf\u0dbb\u0dca\u0d9c\u0dba\u0dba\u0dd2 </li>\n</ul><li><span translate=no>_^_1_^_</span> \u0dc0\u0dda <span translate=no>_^_2_^_</span></li>\n",
|
||||
"Create a dataset for RETRO model training": "RETRO \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1",
|
||||
"Training dataset for RETRO": "RETRO \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"<h1>RETRO training dataset</h1>\n<p>We pre-retrieve nearest neighbors from the <a href=\"database.html\">key-value database</a> and create the dataset to train the <a href=\"index.html\">RETRO</a> <a href=\"model.html\">model</a>.</p>\n": "<h1>RETRO \u8bad\u7ec3\u6570\u636e\u96c6</h1>\n<p>\u6211\u4eec\u4ece<a href=\"database.html\">\u952e\u503c\u6570\u636e\u5e93</a>\u4e2d\u9884\u5148\u68c0\u7d22\u6700\u8fd1\u7684\u90bb\u57df\uff0c\u5e76\u521b\u5efa\u6570\u636e\u96c6\u6765\u8bad\u7ec3 <a href=\"index.html\">RETRO</a> <a href=\"model.html\">\u6a21\u578b</a>\u3002</p>\n",
|
||||
"<h2>Build the dataset</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the chunk length </li>\n<li><span translate=no>_^_1_^_</span> is the number of chunks per training sample </li>\n<li><span translate=no>_^_2_^_</span> is the maximum number of characters to skip between two samples. We skip a few characters between samples to make sure the samples aren't aligned perfectly with the chunks in the <a href=\"database.html\">database</a></li></ul>\n": "<h2>\u6784\u5efa\u6570\u636e\u96c6</h2>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u533a\u5757\u957f\u5ea6</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u6bcf\u4e2a\u8bad\u7ec3\u6837\u672c\u7684\u5757\u6570</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u5728\u4e24\u4e2a\u6837\u672c\u4e4b\u95f4\u8df3\u8fc7\u7684\u6700\u5927\u5b57\u7b26\u6570\u3002\u6211\u4eec\u5728\u6837\u672c\u4e4b\u95f4\u8df3\u8fc7\u51e0\u4e2a\u5b57\u7b26\uff0c\u4ee5\u786e\u4fdd\u6837\u672c\u4e0e<a href=\"database.html\">\u6570\u636e\u5e93</a>\u4e2d\u7684\u5757\u4e0d\u5b8c\u5168\u5bf9\u9f50</li></ul>\n",
|
||||
"<h2>Dataset</h2>\n<p>This is the PyTorch dataset that loads the dataset created by <span translate=no>_^_0_^_</span>.</p>\n": "<h2>\u6570\u636e\u96c6</h2>\n<p>\u8fd9\u662f PyTorch \u6570\u636e\u96c6\uff0c\u7528\u4e8e\u52a0\u8f7d\u7531\u521b\u5efa\u7684\u6570\u636e\u96c6<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> Get a sample</p>\n": "<p>\u83b7\u53d6\u6837\u54c1</p>\n",
|
||||
"<p> Number of samples</p>\n": "<p>\u6837\u672c\u6570\u91cf</p>\n",
|
||||
"<p>Add to list of samples </p>\n": "<p>\u6dfb\u52a0\u5230\u6837\u54c1\u6e05\u5355</p>\n",
|
||||
"<p>Break it into chunks </p>\n": "<p>\u628a\u5b83\u5206\u6210\u5927\u5757</p>\n",
|
||||
"<p>Collect the offset </p>\n": "<p>\u6536\u96c6\u504f\u79fb\u91cf</p>\n",
|
||||
"<p>Cursor for the text </p>\n": "<p>\u5149\u6807\u6307\u5411\u6587\u672c</p>\n",
|
||||
"<p>For samples </p>\n": "<p>\u5bf9\u4e8e\u6837\u54c1</p>\n",
|
||||
"<p>Get neighbor texts. The neighbor length is twice the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u83b7\u53d6\u90bb\u5c45\u77ed\u4fe1\u3002\u90bb\u5c45\u957f\u5ea6\u662f\u4e24\u500d<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get the sample </p>\n": "<p>\u83b7\u53d6\u6837\u54c1</p>\n",
|
||||
"<p>Get the sample including an extra character (for prediction) </p>\n": "<p>\u83b7\u53d6\u5305\u542b\u989d\u5916\u5b57\u7b26\u7684\u6837\u672c\uff08\u7528\u4e8e\u9884\u6d4b\uff09</p>\n",
|
||||
"<p>Increment the cursor </p>\n": "<p>\u589e\u52a0\u5149\u6807</p>\n",
|
||||
"<p>Iterate through sample offsets </p>\n": "<p>\u904d\u5386\u6837\u672c\u504f\u79fb\u91cf</p>\n",
|
||||
"<p>Load the index for retrieving neighbors </p>\n": "<p>\u52a0\u8f7d\u7d22\u5f15\u4ee5\u68c0\u7d22\u90bb\u5c45</p>\n",
|
||||
"<p>Load the samples </p>\n": "<p>\u52a0\u8f7d\u6837\u54c1</p>\n",
|
||||
"<p>Load the text file </p>\n": "<p>\u52a0\u8f7d\u6587\u672c\u6587\u4ef6</p>\n",
|
||||
"<p>Retrieve nearest neighbors </p>\n": "<p>\u68c0\u7d22\u6700\u8fd1\u7684\u90bb\u5c45</p>\n",
|
||||
"<p>Save the samples in JSON. We don't need to use complex dataset storage mechanisms or pre-tokenize since our dataset is small. </p>\n": "<p>\u4ee5 JSON \u683c\u5f0f\u4fdd\u5b58\u793a\u4f8b\u3002\u6211\u4eec\u4e0d\u9700\u8981\u4f7f\u7528\u590d\u6742\u7684\u6570\u636e\u96c6\u5b58\u50a8\u673a\u5236\u6216\u9884\u6807\u8bb0\u5316\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u6570\u636e\u96c6\u5f88\u5c0f\u3002</p>\n",
|
||||
"<p>Skip a few characters to make sure it's not aligned with the neighbors </p>\n": "<p>\u8df3\u8fc7\u51e0\u4e2a\u89d2\u8272\u4ee5\u786e\u4fdd\u5b83\u4e0d\u4e0e\u90bb\u5c45\u5bf9\u9f50</p>\n",
|
||||
"<p>Stop if we've reached the end of the text </p>\n": "<p>\u5982\u679c\u6211\u4eec\u5230\u8fbe\u4e86\u6587\u5b57\u7684\u672b\u5c3e\uff0c\u5c31\u505c\u4e0b\u6765</p>\n",
|
||||
"<p>The chunk offsets </p>\n": "<p>\u533a\u5757\u504f\u79fb\u91cf</p>\n",
|
||||
"<p>The input </p>\n": "<p>\u8f93\u5165</p>\n",
|
||||
"<p>The input sample offsets </p>\n": "<p>\u8f93\u5165\u6837\u672c\u504f\u79fb</p>\n",
|
||||
"<p>Tokenize </p>\n": "<p>Tokenize</p>\n",
|
||||
"<p>Training portion of it </p>\n": "<p>\u5176\u4e2d\u7684\u4e00\u90e8\u5206\u8bad\u7ec3</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the path of the saved JSON file </li>\n<li><span translate=no>_^_1_^_</span> is the <span translate=no>_^_2_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u4fdd\u5b58\u7684 JSON \u6587\u4ef6\u7684\u8def\u5f84</li>\n</ul><li><span translate=no>_^_1_^_</span>\u662f<span translate=no>_^_2_^_</span></li>\n",
|
||||
"Create a dataset for RETRO model training": "\u521b\u5efa\u7528\u4e8e RETRO \u6a21\u578b\u8bad\u7ec3\u7684\u6570\u636e\u96c6",
|
||||
"Training dataset for RETRO": "RETRO \u7684\u8bad\u7ec3\u6570\u636e\u96c6"
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
{
|
||||
"<h1>RETRO model</h1>\n<p>This is the model definition for <a href=\"index.html\">RETRO</a>.</p>\n": "<h1>\u30ec\u30c8\u30ed\u30e2\u30c7\u30eb</h1>\n<p><a href=\"index.html\">\u3053\u308c\u304cRETRO\u306e\u30e2\u30c7\u30eb\u5b9a\u7fa9\u3067\u3059</a>\u3002</p>\n",
|
||||
"<h2><a href=\"../rope/index.html\">RoPE embeddings</a></h2>\n<p><em>We use rotary position embeddings in self-attention layers. We assume the positional information gets embedded in embeddings and therefore not use them in causal attention. <a href=\"https://arxiv.org/abs/3999902edc8511eba3db37f65e372566\">Non-causal self-attention needs explicit positional information because it cannot infer it</a>.</em></p>\n": "<h2><a href=\"../rope/index.html\">\u30ed\u30fc\u30d7\u57cb\u3081\u8fbc\u307f</a></h2>\n<p><em>\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u306b\u306f\u56de\u8ee2\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059\u3002\u4f4d\u7f6e\u60c5\u5831\u306f\u57cb\u3081\u8fbc\u307f\u306b\u57cb\u3081\u8fbc\u307e\u308c\u3066\u3044\u308b\u305f\u3081\u3001\u56e0\u679c\u95a2\u4fc2\u306b\u306f\u4f7f\u7528\u3057\u306a\u3044\u3068\u60f3\u5b9a\u3057\u3066\u3044\u307e\u3059\u3002<a href=\"https://arxiv.org/abs/3999902edc8511eba3db37f65e372566\">\u56e0\u679c\u95a2\u4fc2\u306e\u306a\u3044\u81ea\u5df1\u6ce8\u610f\u306b\u306f\u3001\u63a8\u6e2c\u3067\u304d\u306a\u3044\u305f\u3081\u3001\u660e\u78ba\u306a\u4f4d\u7f6e\u60c5\u5831\u304c\u5fc5\u8981\u3067\u3059</a></em></p>\u3002\n",
|
||||
"<h2>Chunked Cross-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This is similar to the cross-attention layer defined above.</p>\n<p>This is used in the decoder to pay attention to the retrieved neighbor chunks.</p>\n<p><em>We do not use any explicit positional embeddings here. We assume that the model can represent positional information in the embeddings implicitly.</em></p>\n": "<h2>\u30c1\u30e3\u30f3\u30af\u30fb\u30af\u30ed\u30b9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></h2>\n<p>\u3053\u308c\u306f\u4e0a\u3067\u5b9a\u7fa9\u3057\u305f\u30af\u30ed\u30b9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30ec\u30a4\u30e4\u30fc\u306b\u4f3c\u3066\u3044\u307e\u3059\u3002</p>\n<p>\u3053\u308c\u3092\u30c7\u30b3\u30fc\u30c0\u3067\u4f7f\u7528\u3057\u3066\u3001\u53d6\u5f97\u3057\u305f\u96a3\u63a5\u30c1\u30e3\u30f3\u30af\u306b\u6ce8\u76ee\u3057\u307e\u3059\u3002</p>\n<p><em>\u3053\u3053\u3067\u306f\u660e\u793a\u7684\u306a\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u306f\u4e00\u5207\u4f7f\u7528\u3057\u3066\u3044\u307e\u305b\u3093\u3002\u30e2\u30c7\u30eb\u306f\u57cb\u3081\u8fbc\u307f\u5185\u306e\u4f4d\u7f6e\u60c5\u5831\u3092\u6697\u9ed9\u7684\u306b\u8868\u73fe\u3067\u304d\u308b\u3068\u4eee\u5b9a\u3057\u307e\u3059</em></p>\u3002\n",
|
||||
"<h2>Cross-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This is similar to the self-attention layer defined above, except that it gets keys and values from a different set of embeddings than the queries.</p>\n<p>This is used in the encoder to encode the retrieved chunks based on the input chunks.</p>\n<p><em>We do not use any explicit positional embeddings here. We assume that the model can represent positional information in the embeddings implicitly.</em></p>\n": "<h2>\u30af\u30ed\u30b9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></h2>\n<p>\u3053\u308c\u306f\u4e0a\u3067\u5b9a\u7fa9\u3057\u305f\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u3068\u4f3c\u3066\u3044\u307e\u3059\u304c\u3001\u30af\u30a8\u30ea\u3068\u306f\u7570\u306a\u308b\u57cb\u3081\u8fbc\u307f\u30bb\u30c3\u30c8\u304b\u3089\u30ad\u30fc\u3068\u5024\u3092\u53d6\u5f97\u3059\u308b\u70b9\u304c\u7570\u306a\u308a\u307e\u3059\u3002</p>\n<p>\u3053\u308c\u3092\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u3067\u4f7f\u7528\u3057\u3066\u3001\u53d6\u5f97\u3057\u305f\u30c1\u30e3\u30f3\u30af\u3092\u5165\u529b\u30c1\u30e3\u30f3\u30af\u306b\u57fa\u3065\u3044\u3066\u30a8\u30f3\u30b3\u30fc\u30c9\u3057\u307e\u3059\u3002</p>\n<p><em>\u3053\u3053\u3067\u306f\u660e\u793a\u7684\u306a\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u306f\u4e00\u5207\u4f7f\u7528\u3057\u3066\u3044\u307e\u305b\u3093\u3002\u30e2\u30c7\u30eb\u306f\u57cb\u3081\u8fbc\u307f\u5185\u306e\u4f4d\u7f6e\u60c5\u5831\u3092\u6697\u9ed9\u7684\u306b\u8868\u73fe\u3067\u304d\u308b\u3068\u4eee\u5b9a\u3057\u307e\u3059</em></p>\u3002\n",
|
||||
"<h2>Nearest Neighbor Encoder <span translate=no>_^_0_^_</span></h2>\n<p>This module encodes the retrieved nearest neighbors</p>\n": "<h2>\u8fd1\u508d\u30a8\u30f3\u30b3\u30fc\u30c0 <span translate=no>_^_0_^_</span></h2>\n<p>\u3053\u306e\u30e2\u30b8\u30e5\u30fc\u30eb\u306f\u3001\u53d6\u5f97\u3057\u305f\u6700\u8fd1\u508d\u3092\u30a8\u30f3\u30b3\u30fc\u30c9\u3057\u307e\u3059</p>\n",
|
||||
"<h2>Retro Model</h2>\n<p>This is the Retro decoder</p>\n": "<h2>\u30ec\u30c8\u30ed\u30e2\u30c7\u30eb</h2>\n<p>\u3053\u308c\u306f\u30ec\u30c8\u30ed\u30c7\u30b3\u30fc\u30c0\u30fc\u3067\u3059</p>\n",
|
||||
"<h2>Self-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This applies causal and non-causal <a href=\"../mha.html\">multi-headed self-attention</a>.</p>\n": "<h2>\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></h2>\n<p><a href=\"../mha.html\">\u3053\u308c\u306b\u306f\u3001\u56e0\u679c\u95a2\u4fc2\u3068\u975e\u56e0\u679c\u95a2\u4fc2\u306e\u591a\u9762\u7684\u306a\u81ea\u5df1\u6ce8\u610f\u304c\u5f53\u3066\u306f\u307e\u308a\u307e\u3059\u3002</a></p>\n",
|
||||
"<h3>Mask the attention layer for causal attention</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the attention matrix of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<h3>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u3092\u30de\u30b9\u30af\u3057\u3066\u539f\u56e0\u3068\u306a\u308b\u6ce8\u610f\u3092\u4fc3\u3059</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u30b7\u30a7\u30a4\u30d7\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u3067\u3059 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<h3>Position-wise Feed Forward Layer <span translate=no>_^_0_^_</span></h3>\n<p>This consists of two linear layers and an activation in the middle.</p>\n": "<h3>\u4f4d\u7f6e\u5225\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64 <span translate=no>_^_0_^_</span></h3>\n<p>\u3053\u308c\u306f2\u3064\u306e\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc\u3068\u4e2d\u592e\u306e\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Test the model with fake data</h3>\n": "<h3>\u30d5\u30a7\u30a4\u30af\u30c7\u30fc\u30bf\u3067\u30e2\u30c7\u30eb\u3092\u30c6\u30b9\u30c8</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> are the embeddings of shape <span translate=no>_^_1_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u5f62\u304c\u57cb\u3081\u8fbc\u307e\u308c\u3066\u3044\u308b\u3082\u306e\u3067\u3059 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> are the input embeddings of shape <span translate=no>_^_1_^_</span> <span translate=no>_^_2_^_</span> are the retrieved nearest neighbors of shape <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u56f3\u5f62\u306e\u5165\u529b\u57cb\u3081\u8fbc\u307f\u306f\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u691c\u7d22\u3055\u308c\u305f\u56f3\u5f62\u306e\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u30c7\u30fc\u30bf\u3067\u3059 <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span></p>\n<p>We passed the embeddings of <span translate=no>_^_1_^_</span> to encoder. </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n<p>\u306e\u57cb\u3081\u8fbc\u307f\u3092\u30a8\u30f3\u30b3\u30fc\u30c0\u306b\u6e21\u3057\u307e\u3057\u305f\u3002<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Activation </p>\n": "<p>\u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Add residual connection </p>\n": "<p>\u6b8b\u4f59\u63a5\u7d9a\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Add the residual connection </p>\n": "<p>\u6b8b\u4f59\u63a5\u7d9a\u3092\u8ffd\u52a0</p>\n",
|
||||
"<p>Append <span translate=no>_^_0_^_</span> zero embedding to the left; i.e. right shift it back </p>\n": "<p><span translate=no>_^_0_^_</span>\u5de6\u306b\u30bc\u30ed\u306e\u57cb\u3081\u8fbc\u307f\u3092\u8ffd\u52a0\u3001\u3064\u307e\u308a\u53f3\u306b\u30b7\u30d5\u30c8\u3057\u3066\u623b\u3059</p>\n",
|
||||
"<p>Append empty embeddings to the end to be able to split the input into chunks </p>\n": "<p>\u5165\u529b\u3092\u30c1\u30e3\u30f3\u30af\u306b\u5206\u5272\u3067\u304d\u308b\u3088\u3046\u306b\u3001\u6700\u5f8c\u306b\u7a7a\u306e\u57cb\u3081\u8fbc\u307f\u3092\u8ffd\u52a0\u3057\u307e\u3059</p>\n",
|
||||
"<p>Apply final linear layer. The result will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u5f8c\u306e\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc\u3092\u9069\u7528\u3057\u307e\u3059\u3002\u7d50\u679c\u306f\u5f62\u306b\u306a\u308a\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Apply masks if it's causal attention </p>\n": "<p>\u539f\u56e0\u3068\u306a\u308b\u6ce8\u610f\u304c\u5fc5\u8981\u306a\u5834\u5408\u306f\u30de\u30b9\u30af\u3092\u7740\u7528\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Apply rotary positional embeddings </p>\n": "<p>\u56de\u8ee2\u5f0f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u3092\u9069\u7528</p>\n",
|
||||
"<p>Apply softmax over the last two dimensions <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u5f8c\u306e 2 \u6b21\u5143\u306b\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9\u3092\u9069\u7528 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Attention layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Bi-directional self attention <span translate=no>_^_0_^_</span> </p>\n": "<p>\u53cc\u65b9\u5411\u306e\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Bi-directional self attention layers </p>\n": "<p>\u53cc\u65b9\u5411\u306e\u30bb\u30eb\u30d5\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8a08\u7b97 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate attention probabilities </p>\n": "<p>\u6ce8\u610f\u78ba\u7387\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate attention scores for all chunks. Each retrieved neighbor will pay attention to the original chunk that retrieved it. This will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3059\u3079\u3066\u306e\u30c1\u30e3\u30f3\u30af\u306e\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002\u53d6\u5f97\u3055\u308c\u305f\u5404\u30cd\u30a4\u30d0\u30fc\u306f\u3001\u305d\u308c\u3092\u53d6\u5f97\u3057\u305f\u5143\u306e\u30c1\u30e3\u30f3\u30af\u306b\u6ce8\u76ee\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u5f62\u306b\u306a\u308a\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate attention scores for input chunks. Each chunk will pay attention to neighbors retrieved by the previous chunk. This will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5165\u529b\u30c1\u30e3\u30f3\u30af\u306e\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002\u5404\u30c1\u30e3\u30f3\u30af\u306f\u3001\u524d\u306e\u30c1\u30e3\u30f3\u30af\u3067\u53d6\u5f97\u3057\u305f\u96a3\u63a5\u30c1\u30e3\u30f3\u30af\u306b\u6ce8\u76ee\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u5f62\u306b\u306a\u308a\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate attentions </p>\n": "<p>\u6ce8\u610f\u4e8b\u9805\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate softmax across the last dimension </p>\n": "<p>\u6700\u5f8c\u306e\u6b21\u5143\u306e\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9\u3092\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate the product of position index and <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f4d\u7f6e\u6307\u6570\u306e\u7a4d\u3092\u8a08\u7b97\u3057\u3001<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate</p>\n<span translate=no>_^_0_^_</span><p>for <span translate=no>_^_1_^_</span> </p>\n": "<p>\u8a08\u7b97</p>\n<span translate=no>_^_0_^_</span><p>\u306b\u3068\u3063\u3066 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Causal self attention <span translate=no>_^_0_^_</span> </p>\n": "<p>\u56e0\u679c\u7684\u81ea\u5df1\u6ce8\u610f <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Change from shape <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u5f62\u72b6\u3092\u6b21\u306e\u3088\u3046\u306b\u5909\u66f4 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Chunked cross attention layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u578b\u30af\u30ed\u30b9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Chunked-cross attention if <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u30fb\u30af\u30ed\u30b9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3 if <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Concatenate so that for row <span translate=no>_^_0_^_</span> we have <span translate=no>_^_1_^_</span> </p>\n": "<p>\u884c\u304c\u6b21\u306e\u3088\u3046\u306b\u306a\u308b\u3088\u3046\u306b\u9023\u7d50\u3057\u307e\u3059 <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Create a triangular mask </p>\n": "<p>\u4e09\u89d2\u30de\u30b9\u30af\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create position indexes <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f4d\u7f6e\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u306e\u4f5c\u6210 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Cross attention if <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6b21\u306e\u5834\u5408\u306f\u30af\u30ed\u30b9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Cross-attention layers </p>\n": "<p>\u30af\u30ed\u30b9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Embeddings of the retrieved neighbors <span translate=no>_^_0_^_</span>.</p>\n<p>We use same embeddings for both input and neighbors </p>\n": "<p>\u691c\u7d22\u3057\u305f\u30cd\u30a4\u30d0\u30fc\u306e\u57cb\u3081\u8fbc\u307f\u3002<span translate=no>_^_0_^_</span></p>\n<p>\u5165\u529b\u3068\u8fd1\u508d\u306e\u4e21\u65b9\u306b\u540c\u3058\u57cb\u3081\u8fbc\u307f\u3092\u4f7f\u7528\u3057\u307e\u3059</p>\n",
|
||||
"<p>Extract the shape </p>\n": "<p>\u5f62\u72b6\u3092\u62bd\u51fa</p>\n",
|
||||
"<p>Feed forward layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Feed forward layers </p>\n": "<p>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Feed forward layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Filter by the mask </p>\n": "<p>\u30de\u30b9\u30af\u3067\u7d5e\u308a\u8fbc\u3080</p>\n",
|
||||
"<p>Final linear layer </p>\n": "<p>\u6700\u7d42\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>First linear layer </p>\n": "<p>\u7b2c 1 \u7dda\u5f62\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>For all layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u3059\u3079\u3066\u306e\u30ec\u30a4\u30e4\u30fc\u7528 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Gather values </p>\n": "<p>\u4fa1\u5024\u3092\u96c6\u3081\u308b</p>\n",
|
||||
"<p>Get encoder embeddings before the first <span translate=no>_^_0_^_</span> layer, when <span translate=no>_^_1_^_</span> </p>\n": "<p>\u6b21\u306e\u5834\u5408\u306b\u3001\u6700\u521d\u306e\u30ec\u30a4\u30e4\u30fc\u306e\u524d\u306b\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u306e\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97\u3059\u308b <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get input embeddings <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5165\u529b\u57cb\u3081\u8fbc\u307f\u3092\u53d6\u5f97 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get keys and values from the input chunks </p>\n": "<p>\u5165\u529b\u30c1\u30e3\u30f3\u30af\u304b\u3089\u30ad\u30fc\u3068\u5024\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get keys and values from the retrieved neighbors </p>\n": "<p>\u53d6\u5f97\u3057\u305f\u30cd\u30a4\u30d0\u30fc\u304b\u3089\u30ad\u30fc\u3068\u5024\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get query from the input </p>\n": "<p>\u5165\u529b\u304b\u3089\u30af\u30a8\u30ea\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get query from the retrieved chunks </p>\n": "<p>\u53d6\u5f97\u3057\u305f\u30c1\u30e3\u30f3\u30af\u304b\u3089\u30af\u30a8\u30ea\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get query, key, and values and split them in to heads. These will have shapes <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30af\u30a8\u30ea\u3001\u30ad\u30fc\u3001\u5024\u3092\u53d6\u5f97\u3057\u3001\u305d\u308c\u3089\u3092\u30d8\u30c3\u30c9\u306b\u5206\u5272\u3057\u307e\u3059\u3002\u3053\u308c\u3089\u306b\u306f\u5f62\u304c\u3042\u308a\u307e\u3059 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get shape </p>\n": "<p>\u30b7\u30a7\u30a4\u30d7\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get values </p>\n": "<p>\u5024\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Increment chunked cross-attention index </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u30fb\u30af\u30ed\u30b9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u5897\u3084\u3059</p>\n",
|
||||
"<p>Incremnt the cross attention index </p>\n": "<p>\u30af\u30ed\u30b9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u5897\u3084\u3059</p>\n",
|
||||
"<p>Keep index of the chunked cross attention layer </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u5316\u3055\u308c\u305f\u30af\u30ed\u30b9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30ec\u30a4\u30e4\u30fc\u306e\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u4fdd\u6301</p>\n",
|
||||
"<p>Keep the index of the cross attention layer </p>\n": "<p>\u30af\u30ed\u30b9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30ec\u30a4\u30e4\u30fc\u306e\u30a4\u30f3\u30c7\u30c3\u30af\u30b9\u3092\u7dad\u6301</p>\n",
|
||||
"<p>Linear layers for query, key and value heads. </p>\n": "<p>\u30af\u30a8\u30ea\u3001\u30ad\u30fc\u3001\u30d0\u30ea\u30e5\u30fc\u30d8\u30c3\u30c9\u7528\u306e\u30ea\u30cb\u30a2\u30ec\u30a4\u30e4\u30fc\u3002</p>\n",
|
||||
"<p>No attention if there are no chunks (for short inputs when sampling) </p>\n": "<p>\u30c1\u30e3\u30f3\u30af\u304c\u306a\u3044\u5834\u5408\u306f\u4e0d\u8981 (\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u6642\u306e\u5165\u529b\u304c\u77ed\u3044\u5834\u5408)</p>\n",
|
||||
"<p>No masking for non-causal attention </p>\n": "<p>\u56e0\u679c\u95a2\u4fc2\u306e\u306a\u3044\u6ce8\u610f\u306e\u305f\u3081\u306e\u30de\u30b9\u30ad\u30f3\u30b0\u306a\u3057</p>\n",
|
||||
"<p>Normalize encoder embeddings </p>\n": "<p>\u30a8\u30f3\u30b3\u30fc\u30c0\u57cb\u3081\u8fbc\u307f\u3092\u6b63\u898f\u5316</p>\n",
|
||||
"<p>Normalize retrieved chunks </p>\n": "<p>\u53d6\u5f97\u3057\u305f\u30c1\u30e3\u30f3\u30af\u3092\u6b63\u898f\u5316</p>\n",
|
||||
"<p>Pre-norm </p>\n": "<p>\u30d7\u30ec\u30ce\u30eb\u30e0</p>\n",
|
||||
"<p>Pre-norm layer </p>\n": "<p>\u30d7\u30ec\u30fb\u30ce\u30eb\u30e0\u30fb\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Pre-norm layer for the query embeddings. The paper uses RMSNorm instead. </p>\n": "<p>\u30af\u30a8\u30ea\u57cb\u3081\u8fbc\u307f\u7528\u306e\u30d7\u30ec\u30ce\u30eb\u30e0\u30ec\u30a4\u30e4\u30fc\u3002\u3053\u306e\u8ad6\u6587\u3067\u306f\u4ee3\u308f\u308a\u306bRMSnorm\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</p>\u3002\n",
|
||||
"<p>Pre-norm layer. The paper uses RMSNorm instead. </p>\n": "<p>\u30d7\u30ec\u30fb\u30ce\u30eb\u30e0\u30fb\u30ec\u30a4\u30e4\u30fc\u3053\u306e\u8ad6\u6587\u3067\u306f\u4ee3\u308f\u308a\u306bRMSnorm\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</p>\u3002\n",
|
||||
"<p>Pre-normalization </p>\n": "<p>\u4e8b\u524d\u6b63\u898f\u5316</p>\n",
|
||||
"<p>Pre-normalization layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u306e\u4e8b\u524d\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Pre-normalization layer for nearest neighbor embeddings from <span translate=no>_^_0_^_</span> </p>\n": "<p>\u304b\u3089\u306e\u6700\u8fd1\u508d\u57cb\u3081\u8fbc\u307f\u7528\u306e\u4e8b\u524d\u6b63\u898f\u5316\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>ReLU Activation </p>\n": "<p>ReLU \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Readout layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8aad\u307f\u51fa\u3057\u5c64 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Remove the first <span translate=no>_^_0_^_</span> embeddings. The input pays attention to neighbors retrieved and encoded using the past tokens only; so that there is no information leakage. That is the retrieved neighbors from the first chunks will have information from the first chunk. So by shifting the sequence to the left by <span translate=no>_^_1_^_</span> we make sure that information only flows to the right. </p>\n": "<p><span translate=no>_^_0_^_</span>\u6700\u521d\u306e\u57cb\u3081\u8fbc\u307f\u3092\u524a\u9664\u3057\u307e\u3059\u3002\u5165\u529b\u306f\u3001\u904e\u53bb\u306e\u30c8\u30fc\u30af\u30f3\u306e\u307f\u3092\u4f7f\u7528\u3057\u3066\u53d6\u5f97\u304a\u3088\u3073\u30a8\u30f3\u30b3\u30fc\u30c9\u3055\u308c\u305f\u30cd\u30a4\u30d0\u30fc\u306b\u6ce8\u76ee\u3057\u3001\u60c5\u5831\u6f0f\u3048\u3044\u304c\u767a\u751f\u3057\u306a\u3044\u3088\u3046\u306b\u3057\u307e\u3059\u3002\u3064\u307e\u308a\u3001\u6700\u521d\u306e\u30c1\u30e3\u30f3\u30af\u304b\u3089\u53d6\u5f97\u3057\u305f\u30cd\u30a4\u30d0\u30fc\u306b\u306f\u3001\u6700\u521d\u306e\u30c1\u30e3\u30f3\u30af\u304b\u3089\u306e\u60c5\u5831\u304c\u542b\u307e\u308c\u307e\u3059\u3002\u305d\u306e\u305f\u3081\u3001\u30b7\u30fc\u30b1\u30f3\u30b9\u3092\u5de6\u306b\u30b7\u30d5\u30c8\u3059\u308b\u3053\u3068\u3067<span translate=no>_^_1_^_</span>\u3001\u60c5\u5831\u304c\u53f3\u306b\u306e\u307f\u6d41\u308c\u308b\u3088\u3046\u306b\u3057\u307e\u3059</p>\u3002\n",
|
||||
"<p>Reshape the input into chunks. </p>\n": "<p>\u5165\u529b\u3092\u30c1\u30e3\u30f3\u30af\u306b\u30ea\u30b7\u30a7\u30a4\u30d7\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Residual </p>\n": "<p>\u6b8b\u4f59</p>\n",
|
||||
"<p>Residual connection </p>\n": "<p>\u6b8b\u7559\u63a5\u7d9a</p>\n",
|
||||
"<p>Rotary positional embeddings </p>\n": "<p>\u30ed\u30fc\u30bf\u30ea\u30fc\u30dd\u30b8\u30b7\u30e7\u30f3\u30a8\u30f3\u30d9\u30c7\u30a3\u30f3\u30b0</p>\n",
|
||||
"<p>Scale attention scores </p>\n": "<p>\u30b9\u30b1\u30fc\u30eb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2</p>\n",
|
||||
"<p>Scale it by <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u30fb\u30d0\u30a4\u30fb\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Second linear layer </p>\n": "<p>2 \u756a\u76ee\u306e\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Softmax for attention probabilities </p>\n": "<p>\u6ce8\u610f\u78ba\u7387\u306e\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9</p>\n",
|
||||
"<p>The two linear layers </p>\n": "<p>2 \u3064\u306e\u7dda\u5f62\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>To scale attentions before softmax by <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9\u306e\u524d\u306b\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u3092\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3059\u308b\u306b\u306f <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Token embedding layer </p>\n": "<p>\u30c8\u30fc\u30af\u30f3\u57cb\u3081\u8fbc\u307f\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Truncate and add the residual connection </p>\n": "<p>\u6b8b\u308a\u306e\u63a5\u7d9a\u3092\u5207\u308a\u6368\u3066\u3066\u8ffd\u52a0</p>\n",
|
||||
"<p>return <span translate=no>_^_0_^_</span> </p>\n": "<p>\u623b\u308b <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are the retrieved nearest neighbor chunk embeddings with shape <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> are the input chunks from which the nearest neighbors were retrieved with shape <span translate=no>_^_3_^_</span>. This is already normalized.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u691c\u7d22\u3055\u308c\u305f\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u306e\u30c1\u30e3\u30f3\u30af\u57cb\u3081\u8fbc\u307f\u306e\u5f62\u72b6\u306e\u57cb\u3081\u8fbc\u307f\u3067\u3059 <span translate=no>_^_1_^_</span></li>\n<li><span translate=no>_^_2_^_</span>\u306f\u5165\u529b\u30c1\u30e3\u30f3\u30af\u3067\u3001\u305d\u3053\u304b\u3089\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u30c7\u30fc\u30bf\u304c\u53d6\u5f97\u3055\u308c\u307e\u3057\u305f\u3002<span translate=no>_^_3_^_</span>\u3053\u308c\u306f\u3059\u3067\u306b\u6a19\u6e96\u5316\u3055\u308c\u3066\u3044\u307e\u3059\u3002</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are token embeddings of the retrieved nearest neighbors, <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span></li></ul>\n<ul><li><span translate=no>_^_3_^_</span> is are the input token embeddings, <span translate=no>_^_4_^_</span> of shape <span translate=no>_^_5_^_</span></li></ul>\n<p><em>The chunks <span translate=no>_^_6_^_</span> and neighbors <span translate=no>_^_7_^_</span> are processed in parallel.</em></p>\n": "<ul><li><span translate=no>_^_0_^_</span>\u691c\u7d22\u3055\u308c\u305f\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u306e\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3067\u3001\u5f62\u3082\u6574\u3063\u3066\u3044\u307e\u3059 <span translate=no>_^_1_^_</span> <span translate=no>_^_2_^_</span></li></ul>\n<ul><li><span translate=no>_^_3_^_</span>\u306f\u5165\u529b\u30c8\u30fc\u30af\u30f3\u306e\u57cb\u3081\u8fbc\u307f\u3067\u3001\u5f62\u72b6\u306f\u3055\u307e\u3056\u307e\u3067\u3059 <span translate=no>_^_4_^_</span> <span translate=no>_^_5_^_</span></li></ul>\n<p><em><span translate=no>_^_6_^_</span><span translate=no>_^_7_^_</span>\u30c1\u30e3\u30f3\u30af\u3068\u30cd\u30a4\u30d0\u30fc\u306f\u4e26\u884c\u3057\u3066\u51e6\u7406\u3055\u308c\u307e\u3059\u3002</em></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the Tensor at the head of a key or a query with shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30ad\u30fc\u307e\u305f\u306f\u5f62\u72b6\u306e\u3042\u308b\u30af\u30a8\u30ea\u306e\u5148\u982d\u306b\u3042\u308b\u30c6\u30f3\u30bd\u30eb\u3067\u3059 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input sequence, <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> are the retrieved neighbors <span translate=no>_^_4_^_</span> of shape <span translate=no>_^_5_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u306f\u5f62\u72b6\u306e\u5165\u529b\u30b7\u30fc\u30b1\u30f3\u30b9 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u691c\u7d22\u3055\u308c\u305f\u30b7\u30a7\u30a4\u30d7\u306e\u8fd1\u508d\u3067\u3059 <span translate=no>_^_4_^_</span> <span translate=no>_^_5_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the length of a chunk </li>\n<li><span translate=no>_^_1_^_</span> is the number of layers in the encoder <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> are the layers with cross attention <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the number of features in embeddings </li>\n<li><span translate=no>_^_6_^_</span> is the number of heads in attention layers </li>\n<li><span translate=no>_^_7_^_</span> is the size of attention heads </li>\n<li><span translate=no>_^_8_^_</span> is the size of the feed-forward networks hidden layers</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30c1\u30e3\u30f3\u30af\u306e\u9577\u3055\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u306e\u30ec\u30a4\u30e4\u30fc\u6570\u3067\u3059 <span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u30af\u30ed\u30b9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u306e\u5bfe\u8c61\u3068\u306a\u308b\u30ec\u30a4\u30e4\u30fc\u306f <span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u306f\u57cb\u3081\u8fbc\u307f\u306b\u542b\u307e\u308c\u308b\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_6_^_</span>\u306f\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u306e\u30d8\u30c3\u30c9\u6570\u3067\u3059</li>\n<li><span translate=no>_^_7_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u30b5\u30a4\u30ba\u3067\u3059</li>\n<li><span translate=no>_^_8_^_</span>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u96a0\u308c\u5c64\u306e\u30b5\u30a4\u30ba</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the constant used for calculating <span translate=no>_^_3_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u306f\u6a5f\u80fd\u306e\u6570 <span translate=no>_^_1_^_</span></li>\n<li><span translate=no>_^_2_^_</span>\u306f\u8a08\u7b97\u306b\u4f7f\u7528\u3055\u308c\u308b\u5b9a\u6570\u3067\u3059 <span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number features in the hidden layer</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u57cb\u3081\u8fbc\u307f\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u30fc\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u96a0\u308c\u30ec\u30a4\u30e4\u30fc\u306e\u30ca\u30f3\u30d0\u30fc\u30d5\u30a3\u30fc\u30c1\u30e3\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head </li>\n<li><span translate=no>_^_3_^_</span> indicates whether this is causal attention (masked)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u57cb\u3081\u8fbc\u307f\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u30fc\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30d8\u30c3\u30c9\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d8\u30c3\u30c9\u3042\u305f\u308a\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u6570\u3067\u3059</li>\n<li><span translate=no>_^_3_^_</span>\u3053\u308c\u304c\u56e0\u679c\u95a2\u4fc2\u3067\u3042\u308b\u304b\u3069\u3046\u304b\u3092\u793a\u3057\u307e\u3059 (\u30de\u30b9\u30af\u3055\u308c\u3066\u3044\u307e\u3059)</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head </li>\n<li><span translate=no>_^_3_^_</span> is the length of a chunk</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u57cb\u3081\u8fbc\u307f\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u30fc\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30d8\u30c3\u30c9\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d8\u30c3\u30c9\u3042\u305f\u308a\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u6570\u3067\u3059</li>\n<li><span translate=no>_^_3_^_</span>\u30c1\u30e3\u30f3\u30af\u306e\u9577\u3055\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u57cb\u3081\u8fbc\u307f\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u30fc\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30fb\u30d8\u30c3\u30c9\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30d8\u30c3\u30c9\u3042\u305f\u308a\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u6570\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of tokens in the vocabulary </li>\n<li><span translate=no>_^_1_^_</span> is the number of features in embeddings </li>\n<li><span translate=no>_^_2_^_</span> is the number of layers in the decoder <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> are the layers with cross attention <span translate=no>_^_5_^_</span> </li>\n<li><span translate=no>_^_6_^_</span> is the length of a chunk </li>\n<li><span translate=no>_^_7_^_</span> is the number of heads in attention layers </li>\n<li><span translate=no>_^_8_^_</span> is the size of attention heads </li>\n<li><span translate=no>_^_9_^_</span> is the size of the feed-forward networks hidden layers </li>\n<li><span translate=no>_^_10_^_</span> is the nearest neighbor encoder</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u5185\u306e\u30c8\u30fc\u30af\u30f3\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u306f\u57cb\u3081\u8fbc\u307f\u306b\u542b\u307e\u308c\u308b\u30d5\u30a3\u30fc\u30c1\u30e3\u306e\u6570\u3067\u3059</li>\n<li><span translate=no>_^_2_^_</span>\u306f\u30c7\u30b3\u30fc\u30c0\u30fc\u306e\u5c64\u6570\u3067\u3059 <span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u30af\u30ed\u30b9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u306e\u5bfe\u8c61\u3068\u306a\u308b\u30ec\u30a4\u30e4\u30fc\u306f <span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span>\u30c1\u30e3\u30f3\u30af\u306e\u9577\u3055\u3067\u3059</li>\n<li><span translate=no>_^_7_^_</span>\u306f\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u306e\u30d8\u30c3\u30c9\u6570\u3067\u3059</li>\n<li><span translate=no>_^_8_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d8\u30c3\u30c9\u306e\u30b5\u30a4\u30ba\u3067\u3059</li>\n<li><span translate=no>_^_9_^_</span>\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u96a0\u308c\u5c64\u306e\u30b5\u30a4\u30ba</li>\n<li><span translate=no>_^_10_^_</span>\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer embeddings of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u5f62\u3092\u3057\u305f\u5909\u5727\u5668\u306e\u57cb\u3081\u8fbc\u307f\u3067\u3059 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"RETRO model": "\u30ec\u30c8\u30ed\u30e2\u30c7\u30eb",
|
||||
"RETRO model with encoder for neighbors and autoregressive decoder": "\u8fd1\u508d\u7528\u306e\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u3068\u81ea\u5df1\u56de\u5e30\u30c7\u30b3\u30fc\u30c0\u30fc\u3092\u5099\u3048\u305f RETRO \u30e2\u30c7\u30eb"
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
{
|
||||
"<h1>RETRO model</h1>\n<p>This is the model definition for <a href=\"index.html\">RETRO</a>.</p>\n<p><a href=\"https://app.labml.ai/run/3113dd3ea1e711ec85ee295d18534021\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h1>\n<p><a href=\"index.html\">RETRO</a>\u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0d85\u0dbb\u0dca\u0dae \u0daf\u0dd0\u0d9a\u0dca\u0dc0\u0dd3\u0db8 \u0db8\u0dd9\u0dba\u0dba\u0dd2. </p>\n<p><a href=\"https://app.labml.ai/run/3113dd3ea1e711ec85ee295d18534021\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2><a href=\"../rope/index.html\">RoPE embeddings</a></h2>\n<p><em>We use rotary position embeddings in self-attention layers. We assume the positional information gets embedded in embeddings and therefore not use them in causal attention. <a href=\"https://arxiv.org/abs/3999902edc8511eba3db37f65e372566\">Non-causal self-attention needs explicit positional information because it cannot infer it</a>.</em></p>\n": "<h2><a href=\"../rope/index.html\">\u0d9a\u0db9\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca</a></h2>\n<p><em>\u0d85\u0db4\u0dd2\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb \u0db7\u0db8\u0dab \u0dad\u0dad\u0dca\u0dad\u0dca\u0dc0\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1. \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dad\u0dd4\u0dc5\u0da7 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dd3 \u0d87\u0dad\u0dd2 \u0d85\u0dad\u0dbb \u0d91\u0db8 \u0db1\u0dd2\u0dc3\u0dcf \u0d92\u0dc0\u0dcf \u0db4\u0ddc\u0daf\u0dd4 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0da7 \u0dbd\u0d9a\u0dca \u0db1\u0ddc\u0d9a\u0dbb\u0db8\u0dd4. <a href=\"https://arxiv.org/abs/3999902edc8511eba3db37f65e372566\">\u0dc4\u0dda\u0dad\u0dd4 \u0db1\u0ddc\u0dc0\u0db1 \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0da7 \u0db4\u0dd0\u0dc4\u0dd0\u0daf\u0dd2\u0dbd\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0db1\u0dca\u0db1\u0dda \u0d91\u0dba\u0da7 \u0d85\u0db1\u0dd4\u0db8\u0dcf\u0db1 \u0d9a\u0dc5 \u0db1\u0ddc\u0dc4\u0dd0\u0d9a\u0dd2 \u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dd2</a>. </em></p>\n",
|
||||
"<h2>Chunked Cross-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This is similar to the cross-attention layer defined above.</p>\n<p>This is used in the decoder to pay attention to the retrieved neighbor chunks.</p>\n<p><em>We do not use any explicit positional embeddings here. We assume that the model can represent positional information in the embeddings implicitly.</em></p>\n": "<h2>\u0dad\u0dd0\u0dc5\u0dd4\u0dab\u0dd4\u0dc4\u0dbb\u0dc3\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span></h2>\n<p>\u0db8\u0dd9\u0dba\u0d89\u0dc4\u0dad \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0dc4\u0dbb\u0dc3\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0da7 \u0dc3\u0db8\u0dcf\u0db1 \u0dc0\u0dda. </p>\n<p>\u0db8\u0dd9\u0dba\u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc0\u0dd9\u0dad \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba \u0dad\u0dd4\u0dc5 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0dc0\u0dda. </p>\n<p><em>\u0d85\u0db4\u0dd2\u0db8\u0dd9\u0dc4\u0dd2 \u0d9a\u0dd2\u0dc3\u0dd2\u0daf\u0dd4 \u0db4\u0dd0\u0dc4\u0dd0\u0daf\u0dd2\u0dbd\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0db8\u0dd4. \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0da7 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 \u0dc0\u0dca\u0dba\u0d82\u0d9c\u0dba\u0dd9\u0db1\u0dca \u0db1\u0dd2\u0dba\u0ddd\u0da2\u0db1\u0dba \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0dba\u0dd0\u0dba\u0dd2 \u0d85\u0db4\u0dd2 \u0d8b\u0db4\u0d9a\u0dbd\u0dca\u0db4\u0db1\u0dba \u0d9a\u0dbb\u0db8\u0dd4. </em></p>\n",
|
||||
"<h2>Cross-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This is similar to the self-attention layer defined above, except that it gets keys and values from a different set of embeddings than the queries.</p>\n<p>This is used in the encoder to encode the retrieved chunks based on the input chunks.</p>\n<p><em>We do not use any explicit positional embeddings here. We assume that the model can represent positional information in the embeddings implicitly.</em></p>\n": "<h2>\u0dc4\u0dbb\u0dc3\u0dca\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span></h2>\n<p>\u0db8\u0dd9\u0dba\u0d89\u0dc4\u0dad \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad\u0dd2 \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0da7 \u0dc3\u0db8\u0dcf\u0db1 \u0dc0\u0dda, \u0d91\u0dba \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0dca \u0dc0\u0dbd\u0da7 \u0dc0\u0da9\u0dcf \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0dba\u0dad\u0dd4\u0dbb\u0dd4 \u0dc3\u0dc4 \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dd3. </p>\n<p>\u0d86\u0daf\u0dcf\u0db1\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca\u0dc0 \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0d9a\u0dda\u0dad\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dd9\u0dba \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba\u0dda \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0dc0\u0dda. </p>\n<p><em>\u0d85\u0db4\u0dd2\u0db8\u0dd9\u0dc4\u0dd2 \u0d9a\u0dd2\u0dc3\u0dd2\u0daf\u0dd4 \u0db4\u0dd0\u0dc4\u0dd0\u0daf\u0dd2\u0dbd\u0dd2 \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0d9a\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0db8\u0dd4. \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0da7 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 \u0dc0\u0dca\u0dba\u0d82\u0d9c\u0dba\u0dd9\u0db1\u0dca \u0db1\u0dd2\u0dba\u0ddd\u0da2\u0db1\u0dba \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0dba\u0dd0\u0dba\u0dd2 \u0d85\u0db4\u0dd2 \u0d8b\u0db4\u0d9a\u0dbd\u0dca\u0db4\u0db1\u0dba \u0d9a\u0dbb\u0db8\u0dd4. </em></p>\n",
|
||||
"<h2>Nearest Neighbor Encoder <span translate=no>_^_0_^_</span></h2>\n<p>This module encodes the retrieved nearest neighbors</p>\n": "<h2>\u0dc5\u0d9f\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2 \u0d86\u0d9a\u0dda\u0dad\u0d9a\u0dba <span translate=no>_^_0_^_</span></h2>\n<p>\u0db8\u0dd9\u0db8\u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0d86\u0dc3\u0db1\u0dca\u0db1\u0dad\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dc3\u0d82\u0d9a\u0dda\u0dad\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0dba\u0dd2</p>\n",
|
||||
"<h2>Retro Model</h2>\n<p>This is the Retro decoder</p>\n": "<h2>\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n<p>\u0db8\u0dd9\u0dba\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd \u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba</p>\n",
|
||||
"<h2>Self-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This applies causal and non-causal <a href=\"../mha.html\">multi-headed self-attention</a>.</p>\n": "<h2>\u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span></h2>\n<p>\u0db8\u0dd9\u0dba\u0dc4\u0dda\u0dad\u0dd4 \u0dc3\u0dc4 \u0dc4\u0dda\u0dad\u0dd4 \u0db1\u0ddc\u0dc0\u0db1 <a href=\"../mha.html\">\u0db6\u0dc4\u0dd4-\u0dc4\u0dd2\u0dc3 \u0dc3\u0dc4\u0dd2\u0dad \u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba</a>\u0d85\u0daf\u0dcf\u0dc5 \u0dc0\u0dda. </p>\n",
|
||||
"<h3>Mask the attention layer for causal attention</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the attention matrix of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<h3>\u0dc4\u0dda\u0dad\u0dd4\u0d9a\u0dcf\u0dbb\u0d9a\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dc3\u0dca\u0dad\u0dbb\u0dba \u0d86\u0dc0\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d85\u0db1\u0dd4\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dd2 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<h3>Position-wise Feed Forward Layer <span translate=no>_^_0_^_</span></h3>\n<p>This consists of two linear layers and an activation in the middle.</p>\n": "<h3>\u0dc3\u0dca\u0dae\u0dcf\u0db1-\u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca\u0db4\u0ddd\u0dc2\u0dab\u0dba \u0d89\u0daf\u0dd2\u0dbb\u0dd2 \u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span></h3>\n<p>\u0db8\u0dd9\u0dba\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb \u0daf\u0dd9\u0d9a\u0d9a\u0dca \u0dc3\u0dc4 \u0db8\u0dd0\u0daf \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2\u0db1\u0dca \u0dc3\u0db8\u0db1\u0dca\u0dc0\u0dd2\u0dad \u0dc0\u0dda. </p>\n",
|
||||
"<h3>Test the model with fake data</h3>\n": "<h3>\u0dc0\u0dca\u0dba\u0dcf\u0da2\u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0d9f \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db4\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> are the embeddings of shape <span translate=no>_^_1_^_</span></p>\n": "<p> <span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dda <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> are the input embeddings of shape <span translate=no>_^_1_^_</span> <span translate=no>_^_2_^_</span> are the retrieved nearest neighbors of shape <span translate=no>_^_3_^_</span></p>\n": "<p> <span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d86\u0daf\u0dcf\u0db1 <span translate=no>_^_1_^_</span> <span translate=no>_^_2_^_</span> \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dba\u0db1\u0dd4 \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0d86\u0dc3\u0db1\u0dca\u0db1\u0dad\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2\u0dba <span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span></p>\n<p>We passed the embeddings of <span translate=no>_^_1_^_</span> to encoder. </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n<p>\u0d85\u0db4\u0dd2 <span translate=no>_^_1_^_</span> \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba\u0da7 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0db8\u0dca\u0db8\u0dad \u0d9a\u0dc5\u0dd9\u0db8\u0dd4. </p>\n",
|
||||
"<p>Activation </p>\n": "<p>\u0dc3\u0d9a\u0dca\u200d\u0dbb\u0dd3\u0dba\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Add residual connection </p>\n": "<p>\u0d85\u0dc0\u0dc1\u0dda\u0dc2\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dad\u0dcf\u0dc0\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add the residual connection </p>\n": "<p>\u0d85\u0dc0\u0dc1\u0dda\u0dc2\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dad\u0dcf\u0dc0\u0dba \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Append <span translate=no>_^_0_^_</span> zero embedding to the left; i.e. right shift it back </p>\n": "<p>\u0dc0\u0db8\u0da7 <span translate=no>_^_0_^_</span> \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1; \u0d91\u0db1\u0db8\u0dca \u0daf\u0d9a\u0dd4\u0dab\u0dd4 \u0d91\u0dba \u0d86\u0db4\u0dc3\u0dd4 \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Append empty embeddings to the end to be able to split the input into chunks </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0dba\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc0\u0dbd\u0da7 \u0db6\u0dd9\u0daf\u0dd3\u0db8\u0da7 \u0dc4\u0dd0\u0d9a\u0dd2\u0dc0\u0db1 \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0dc4\u0dd2\u0dc3\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0dba \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Apply final linear layer. The result will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba \u0dba\u0ddc\u0daf\u0db1\u0dca\u0db1. \u0db4\u0dca\u0dbb\u0dad\u0dd2 result \u0dbd\u0dba \u0dc4\u0dd0\u0da9\u0dba \u0d87\u0dad <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Apply masks if it's causal attention </p>\n": "<p>\u0d91\u0dba\u0dc4\u0dda\u0dad\u0dd4 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0d9a\u0dca \u0db1\u0db8\u0dca \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0dba\u0ddc\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Apply rotary positional embeddings </p>\n": "<p>\u0db7\u0dca\u0dbb\u0db8\u0dab\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dba\u0ddc\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Apply softmax over the last two dimensions <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0db8\u0dcf\u0db1\u0dba\u0db1\u0dca \u0daf\u0dd9\u0d9a\u0da7 \u0dc0\u0da9\u0dcf \u0dc3\u0ddc\u0dc6\u0dca\u0da7\u0dca\u0db8\u0dd0\u0d9a\u0dca\u0dc3\u0dca \u0dba\u0ddc\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Attention layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dc3\u0dca\u0dae\u0dbb <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Bi-directional self attention <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0daf\u0dca\u0dc0\u0dd2-\u0daf\u0dd2\u0dc1\u0dcf\u0db1\u0dd4\u0d9c\u0dad\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Bi-directional self attention layers </p>\n": "<p>\u0daf\u0dca\u0dc0\u0dd2-\u0daf\u0dd2\u0dc1\u0dcf\u0db1\u0dd4\u0d9c\u0dad\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb </p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate attention probabilities </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate attention scores for all chunks. Each retrieved neighbor will pay attention to the original chunk that retrieved it. This will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dbd\u0d9a\u0dd4\u0dab\u0dd4 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1. \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0dc3\u0dd1\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dd9\u0d9a\u0dd4\u0db8 \u0d91\u0dba \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0db8\u0dd4\u0dbd\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad. \u0db8\u0dd9\u0db8 \u0dc4\u0dd0\u0da9\u0dba \u0d87\u0dad <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate attention scores for input chunks. Each chunk will pay attention to neighbors retrieved by the previous chunk. This will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dbd\u0d9a\u0dd4\u0dab\u0dd4 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1. \u0dc3\u0dd1\u0db8 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a\u0dca\u0db8 \u0d9a\u0dbd\u0dd2\u0db1\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba \u0dc0\u0dd2\u0dc3\u0dd2\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1\u0dcf \u0dbd\u0daf \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dd4 \u0d87\u0dad. \u0db8\u0dd9\u0db8 \u0dc4\u0dd0\u0da9\u0dba \u0d87\u0dad <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate attentions </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate softmax across the last dimension </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0db8\u0dcf\u0db1\u0dba \u0dc4\u0dbb\u0dc4\u0dcf \u0dc3\u0ddc\u0dc6\u0dca\u0da7\u0dca\u0db8\u0dd0\u0d9a\u0dca\u0dc3\u0dca \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate the product of position index and <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba\u0dda \u0db1\u0dd2\u0dc2\u0dca\u0db4\u0dcf\u0daf\u0dd2\u0dad\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate</p>\n<span translate=no>_^_0_^_</span><p>for <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0d9c\u0dab\u0db1\u0dba\u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n<span translate=no>_^_0_^_</span><p>\u0dc3\u0db3\u0dc4\u0dcf <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Causal self attention <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc4\u0dda\u0dad\u0dd4\u0d9a\u0dcf\u0dbb\u0d9a\u0dc3\u0dca\u0dc0\u0dba\u0d82 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Change from shape <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca\u0dc0\u0dd9\u0db1\u0dc3\u0dca <span translate=no>_^_0_^_</span> \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Chunked cross attention layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0db4\u0db1\u0dbd\u0daf \u0dc4\u0dbb\u0dc3\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Chunked-cross attention if <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dd4\u0dbb\u0dd4\u0dc3-\u0dc4\u0dbb\u0dc3\u0dca\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db1\u0db8\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Concatenate so that for row <span translate=no>_^_0_^_</span> we have <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0db4\u0dda\u0dc5\u0dd2\u0dba\u0dc3\u0db3\u0dc4\u0dcf <span translate=no>_^_0_^_</span> \u0d85\u0db4\u0da7 \u0d87\u0dad\u0dd2 \u0db4\u0dbb\u0dd2\u0daf\u0dd2 \u0dc3\u0d82\u0dba\u0dd4\u0d9a\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Create a triangular mask </p>\n": "<p>\u0dad\u0dca\u0dbb\u0dd2\u0d9a\u0ddd\u0dab\u0dcf\u0d9a\u0dcf\u0dbb\u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create position indexes <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0daf\u0dbb\u0dca\u0dc1\u0d9a \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Cross attention if <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc4\u0dbb\u0dc3\u0dca\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dda \u0db1\u0db8\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Cross-attention layers </p>\n": "<p>\u0dc4\u0dbb\u0dc3\u0dca\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb </p>\n",
|
||||
"<p>Embeddings of the retrieved neighbors <span translate=no>_^_0_^_</span>.</p>\n<p>We use same embeddings for both input and neighbors </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0dad\u0dca \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dcf\u0dc3\u0dd3\u0db1\u0dca\u0d9c\u0dda \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca <span translate=no>_^_0_^_</span>. </p>\n<p>\u0d86\u0daf\u0dcf\u0db1\u0dc3\u0dc4 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 \u0d91\u0d9a\u0db8 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
|
||||
"<p>Extract the shape </p>\n": "<p>\u0dc4\u0dd0\u0da9\u0dba\u0d8b\u0db4\u0dd4\u0da7\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Feed forward layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d89\u0daf\u0dd2\u0dbb\u0dd2\u0dc3\u0dca\u0dae\u0dbb\u0dba \u0db4\u0ddd\u0dc2\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Feed forward layers </p>\n": "<p>\u0d89\u0daf\u0dd2\u0dbb\u0dd2\u0dc3\u0dca\u0dae\u0dbb \u0db4\u0ddd\u0dc2\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Feed forward layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d89\u0daf\u0dd2\u0dbb\u0dd2\u0dc3\u0dca\u0dae\u0dbb \u0db4\u0ddd\u0dc2\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Filter by the mask </p>\n": "<p>\u0dc0\u0dd9\u0dc3\u0dca\u0db8\u0dd4\u0dc4\u0dd4\u0dab\u0db8\u0d9c\u0dd2\u0db1\u0dca \u0db4\u0dd9\u0dbb\u0dc4\u0db1\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Final linear layer </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>First linear layer </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>For all layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dd2\u0dba\u0dbd\u0dd4\u0db8\u0dc3\u0dca\u0dae\u0dbb \u0dc3\u0db3\u0dc4\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Gather values </p>\n": "<p>\u0dc0\u0da7\u0dd2\u0db1\u0dcf\u0d9a\u0db8\u0dca\u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get encoder embeddings before the first <span translate=no>_^_0_^_</span> layer, when <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4 <span translate=no>_^_0_^_</span> \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0da7 \u0db4\u0dd9\u0dbb \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Get input embeddings <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get keys and values from the input chunks </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0dba\u0dad\u0dd4\u0dbb\u0dd4 \u0dc3\u0dc4 \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get keys and values from the retrieved neighbors </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0dad\u0dca \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca\u0d9c\u0dd9\u0db1\u0dca \u0dba\u0dad\u0dd4\u0dbb\u0dd4 \u0dc3\u0dc4 \u0dc0\u0da7\u0dd2\u0db1\u0dcf\u0d9a\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get query from the input </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0dba\u0dd9\u0db1\u0dca\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8 \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get query from the retrieved chunks </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0dad\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8 \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get query, key, and values and split them in to heads. These will have shapes <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8, \u0dba\u0dad\u0dd4\u0dbb \u0dc3\u0dc4 \u0d85\u0d9c\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dd9\u0db1 \u0d92\u0dc0\u0dcf \u0dc4\u0dd2\u0dc3\u0dca \u0dc0\u0dbd\u0da7 \u0db6\u0dd9\u0daf\u0db1\u0dca\u0db1. \u0db8\u0dda\u0dc0\u0dcf\u0da7 \u0dc4\u0dd0\u0da9\u0dba\u0db1\u0dca \u0d87\u0dad <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Get shape </p>\n": "<p>\u0dc4\u0dd0\u0da9\u0dba\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get values </p>\n": "<p>\u0dc0\u0da7\u0dd2\u0db1\u0dcf\u0d9a\u0db8\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Increment chunked cross-attention index </p>\n": "<p>\u0dc0\u0dbb\u0dca\u0db0\u0d9a\u0d9a\u0db4\u0db1 \u0dbd\u0daf \u0dc4\u0dbb\u0dc3\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba </p>\n",
|
||||
"<p>Incremnt the cross attention index </p>\n": "<p>\u0dc4\u0dbb\u0dc3\u0dca\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba \u0dc0\u0dd0\u0da9\u0dd2 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Keep index of the chunked cross attention layer </p>\n": "<p>\u0d9a\u0db4\u0db1\u0dbd\u0daf \u0dc4\u0dbb\u0dc3\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Keep the index of the cross attention layer </p>\n": "<p>\u0dc4\u0dbb\u0dc3\u0dca\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba \u0dad\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Linear layers for query, key and value heads. </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8, \u0dba\u0dad\u0dd4\u0dbb\u0dd4 \u0dc3\u0dc4 \u0d85\u0d9c\u0dba \u0dc4\u0dd2\u0dc3\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb. </p>\n",
|
||||
"<p>No attention if there are no chunks (for short inputs when sampling) </p>\n": "<p>\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0db1\u0ddc\u0db8\u0dd0\u0dad\u0dd2 \u0db1\u0db8\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0db1\u0ddc\u0d9a\u0dbb\u0dba\u0dd2 (\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dda\u0daf\u0dd3 \u0d9a\u0dd9\u0da7\u0dd2 \u0dba\u0dd9\u0daf\u0dc0\u0dd4\u0db8\u0dca \u0dc3\u0db3\u0dc4\u0dcf) </p>\n",
|
||||
"<p>No masking for non-causal attention </p>\n": "<p>\u0dc4\u0dda\u0dad\u0dd4\u0db1\u0ddc\u0dc0\u0db1 \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0dc0\u0dbb\u0dab \u0db1\u0ddc\u0db8\u0dd0\u0dad </p>\n",
|
||||
"<p>Normalize encoder embeddings </p>\n": "<p>\u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Normalize retrieved chunks </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0dad\u0dca \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Pre-norm </p>\n": "<p>\u0db4\u0dd6\u0dbb\u0dca\u0dc0\u0dc3\u0db8\u0dca\u0db8\u0dad\u0dba </p>\n",
|
||||
"<p>Pre-norm layer </p>\n": "<p>\u0db4\u0dd9\u0dbb-\u0dc3\u0db8\u0dca\u0db8\u0dad\u0dc3\u0dca\u0dad\u0dbb\u0dba </p>\n",
|
||||
"<p>Pre-norm layer for the query embeddings. The paper uses RMSNorm instead. </p>\n": "<p>\u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0dc3\u0db8\u0dca\u0db8\u0dad \u0dc3\u0dca\u0dad\u0dbb\u0dba. \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7 RMSNorm \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>Pre-norm layer. The paper uses RMSNorm instead. </p>\n": "<p>\u0db4\u0dd6\u0dbb\u0dca\u0dc0\u0dc3\u0db8\u0dca\u0db8\u0dad \u0dc3\u0dca\u0dad\u0dbb\u0dba. \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 \u0dc0\u0dd9\u0db1\u0dd4\u0dc0\u0da7 RMSNorm \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<p>Pre-normalization </p>\n": "<p>\u0db4\u0dd6\u0dbb\u0dca\u0dc0\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba </p>\n",
|
||||
"<p>Pre-normalization layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0db3\u0dc4\u0dcf\u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab \u0dc3\u0dca\u0dad\u0dbb\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Pre-normalization layer for nearest neighbor embeddings from <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d86\u0dc3\u0db1\u0dca\u0db1\u0dad\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf\u0d9c\u0dda \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd6\u0dbb\u0dca\u0dc0 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab \u0dc3\u0dca\u0dad\u0dbb\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>ReLU Activation </p>\n": "<p>Relu\u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Readout layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dd2\u0dba\u0dc0\u0dd3\u0db8\u0dda\u0dc3\u0dca\u0dae\u0dbb\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Remove the first <span translate=no>_^_0_^_</span> embeddings. The input pays attention to neighbors retrieved and encoded using the past tokens only; so that there is no information leakage. That is the retrieved neighbors from the first chunks will have information from the first chunk. So by shifting the sequence to the left by <span translate=no>_^_1_^_</span> we make sure that information only flows to the right. </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4 <span translate=no>_^_0_^_</span> \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1. \u0d85\u0dad\u0dd3\u0dad \u0da7\u0ddd\u0d9a\u0db1 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dba\u0dd9\u0db1\u0dca \u0db4\u0db8\u0dab\u0d9a\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0dc3\u0dc4 \u0d9a\u0dda\u0dad\u0db1\u0dba \u0d9a\u0dbb \u0d87\u0dad\u0dd2 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0d9a\u0dd9\u0dbb\u0dd9\u0dc4\u0dd2 \u0d86\u0daf\u0dcf\u0db1\u0dba \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2; \u0d91\u0dc0\u0dd2\u0da7 \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 \u0d9a\u0dcf\u0db1\u0dca\u0daf\u0dd4 \u0dc0\u0dd3\u0db8\u0d9a\u0dca \u0dc3\u0dd2\u0daf\u0dd4 \u0db1\u0ddc\u0dc0\u0dda. \u0db4\u0dc5\u0db8\u0dd4 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0dda \u0dc3\u0dd2\u0da7 \u0dbd\u0db6\u0dcf\u0d9c\u0dad\u0dca \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0db4\u0dc5\u0db8\u0dd4 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0dd9\u0db1\u0dca \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 \u0dbd\u0dd0\u0db6\u0dd9\u0db1\u0dd4 \u0d87\u0dad. \u0d91\u0db6\u0dd0\u0dc0\u0dd2\u0db1\u0dca \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba \u0dc0\u0db8\u0da7 \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 \u0db4\u0db8\u0dab\u0d9a\u0dca \u0daf\u0d9a\u0dd4\u0dab\u0da7 \u0d9c\u0dbd\u0dcf \u0dba\u0db1 \u0db6\u0dc0\u0da7 <span translate=no>_^_1_^_</span> \u0d85\u0db4\u0dd2 \u0dc0\u0d9c \u0db6\u0dbd\u0dcf \u0d9c\u0db1\u0dd2\u0db8\u0dd4. </p>\n",
|
||||
"<p>Reshape the input into chunks. </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0dba\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0db6\u0dc0\u0da7 \u0db1\u0dd0\u0dc0\u0dad \u0dc3\u0d9a\u0dc3\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Residual </p>\n": "<p>\u0d85\u0dc0\u0dc1\u0dda\u0dc2 </p>\n",
|
||||
"<p>Residual connection </p>\n": "<p>\u0d85\u0dc0\u0dc1\u0dda\u0dc2\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dad\u0dcf\u0dc0\u0dba </p>\n",
|
||||
"<p>Rotary positional embeddings </p>\n": "<p>\u0dbb\u0ddc\u0da7\u0dbb\u0dd2\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca </p>\n",
|
||||
"<p>Scale attention scores </p>\n": "<p>\u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dbd\u0d9a\u0dd4\u0dab\u0dd4 </p>\n",
|
||||
"<p>Scale it by <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc0\u0dd2\u0dc3\u0dd2\u0db1\u0dca\u0d91\u0dba \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Second linear layer </p>\n": "<p>\u0daf\u0dd9\u0dc0\u0db1\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Softmax for attention probabilities </p>\n": "<p>\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba\u0dc3\u0db8\u0dca\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0dc3\u0ddc\u0dc6\u0dca\u0da7\u0dca\u0db8\u0dd0\u0d9a\u0dca\u0dc3\u0dca </p>\n",
|
||||
"<p>The two linear layers </p>\n": "<p>\u0dbb\u0dda\u0d9b\u0dd3\u0dba\u0dc3\u0dca\u0dae\u0dbb \u0daf\u0dd9\u0d9a </p>\n",
|
||||
"<p>To scale attentions before softmax by <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0ddc\u0dc6\u0dca\u0da7\u0dca\u0db8\u0dd0\u0d9a\u0dca\u0dc3\u0dca\u0dc0\u0dbd\u0da7 \u0db4\u0dd9\u0dbb \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Token embedding layer </p>\n": "<p>\u0da7\u0ddd\u0d9a\u0db1\u0dca\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8 \u0dc3\u0dca\u0dae\u0dbb\u0dba </p>\n",
|
||||
"<p>Truncate and add the residual connection </p>\n": "<p>\u0d85\u0dc0\u0dc1\u0dda\u0dc2\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dad\u0dcf\u0dc0\u0dba \u0d89\u0dc0\u0dad\u0dca \u0d9a\u0dbb \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>return <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d86\u0db4\u0dc3\u0dd4 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are the retrieved nearest neighbor chunk embeddings with shape <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> are the input chunks from which the nearest neighbors were retrieved with shape <span translate=no>_^_3_^_</span>. This is already normalized.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca \u0dba\u0dd4\u0dad\u0dca \u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf\u0d9c\u0dda \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2\u0dba <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> \u0d86\u0dc3\u0db1\u0dca\u0db1\u0dad\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1\u0dcf \u0dbd\u0daf \u0d86\u0daf\u0dcf\u0db1 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 <span translate=no>_^_3_^_</span>\u0dc0\u0dda. \u0db8\u0dd9\u0dba \u0daf\u0dd0\u0db1\u0da7\u0db8\u0dad\u0dca \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0dc0\u0dd3 \u0d87\u0dad. </li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are token embeddings of the retrieved nearest neighbors, <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span></li></ul>\n<ul><li><span translate=no>_^_3_^_</span> is are the input token embeddings, <span translate=no>_^_4_^_</span> of shape <span translate=no>_^_5_^_</span></li></ul>\n<p><em>The chunks <span translate=no>_^_6_^_</span> and neighbors <span translate=no>_^_7_^_</span> are processed in parallel.</em></p>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca\u0d9c\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca, <span translate=no>_^_1_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca <span translate=no>_^_2_^_</span></li></ul>\n<ul><li><span translate=no>_^_3_^_</span> \u0dba\u0db1\u0dd4 \u0d86\u0daf\u0dcf\u0db1 \u0da7\u0ddd\u0d9a\u0db1 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca, <span translate=no>_^_4_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca <span translate=no>_^_5_^_</span></li></ul>\n<p><em>\u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 <span translate=no>_^_6_^_</span> \u0dc3\u0dc4 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dcf\u0dc3\u0dd3\u0db1\u0dca \u0dc3\u0db8\u0dcf\u0db1\u0dca\u0dad\u0dbb\u0dc0 \u0dc3\u0d9a\u0dc3\u0dca <span translate=no>_^_7_^_</span> \u0d9a\u0dbb\u0db1\u0dd4 \u0dbd\u0dd0\u0db6\u0dda. </em></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the Tensor at the head of a key or a query with shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0dba\u0dad\u0dd4\u0dbb\u0d9a \u0dc4\u0dd2\u0dc3\u0dd9\u0dc4\u0dd2 \u0da7\u0dd9\u0db1\u0dca\u0dc3\u0dbb\u0dca \u0dc4\u0ddd \u0dc4\u0dd0\u0da9\u0dba \u0dc3\u0dc4\u0dd2\u0dad \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dd2 <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input sequence, <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> are the retrieved neighbors <span translate=no>_^_4_^_</span> of shape <span translate=no>_^_5_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0d86\u0daf\u0dcf\u0db1 \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba, <span translate=no>_^_1_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> \u0dc4\u0dd0\u0da9\u0dba\u0dd9\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca <span translate=no>_^_4_^_</span> \u0dc0\u0dda <span translate=no>_^_5_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the length of a chunk </li>\n<li><span translate=no>_^_1_^_</span> is the number of layers in the encoder <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> are the layers with cross attention <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the number of features in embeddings </li>\n<li><span translate=no>_^_6_^_</span> is the number of heads in attention layers </li>\n<li><span translate=no>_^_7_^_</span> is the size of attention heads </li>\n<li><span translate=no>_^_8_^_</span> is the size of the feed-forward networks hidden layers</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a \u0daf\u0dd2\u0d9c </li>\n<li><span translate=no>_^_1_^_</span> \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba\u0dda \u0dc3\u0dca\u0dae\u0dbb \u0d9c\u0dab\u0db1 <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> \u0dc4\u0dbb\u0dc3\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0d87\u0dad\u0dd2 \u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dda <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_6_^_</span> \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dbd \u0dc4\u0dd2\u0dc3\u0dca \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_7_^_</span> \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </li>\n<li><span translate=no>_^_8_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0ddd\u0dc2\u0d9a \u0d89\u0daf\u0dd2\u0dbb\u0dd2 \u0da2\u0dcf\u0dbd\u0dba\u0dda \u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dbd \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dba\u0dd2</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the constant used for calculating <span translate=no>_^_3_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0db1\u0dd2\u0dba\u0dad\u0dba <span translate=no>_^_3_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number features in the hidden layer</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 \u0dc0\u0dda </li>\n</ul><li><span translate=no>_^_1_^_</span> \u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dda \u0d85\u0d82\u0d9a \u0dbd\u0d9a\u0dca\u0dc2\u0dab \u0dc0\u0dda</li>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head </li>\n<li><span translate=no>_^_3_^_</span> indicates whether this is causal attention (masked)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 \u0dc0\u0dda </li>\n<li><span translate=no>_^_1_^_</span> \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0 \u0dc0\u0dda </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0dc4\u0dd2\u0dc3\u0d9a\u0da7 \u0d87\u0dad\u0dd2 \u0dbd\u0d9a\u0dca\u0dc2\u0dab \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_3_^_</span> \u0db8\u0dd9\u0dba \u0dc4\u0dda\u0dad\u0dd4\u0d9a\u0dcf\u0dbb\u0d9a \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba (\u0db8\u0dd0\u0dc3\u0dca\u0dc3\u0dd9\u0da9\u0dca) \u0dba\u0db1\u0dca\u0db1 \u0db4\u0dd9\u0db1\u0dca\u0db1\u0dd4\u0db8\u0dca \u0d9a\u0dbb\u0dba\u0dd2</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head </li>\n<li><span translate=no>_^_3_^_</span> is the length of a chunk</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 \u0dc0\u0dda </li>\n<li><span translate=no>_^_1_^_</span> \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0 \u0dc0\u0dda </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0dc4\u0dd2\u0dc3\u0d9a\u0da7 \u0d87\u0dad\u0dd2 \u0dbd\u0d9a\u0dca\u0dc2\u0dab \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_3_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a \u0daf\u0dd2\u0d9c</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 \u0dc0\u0dda </li>\n<li><span translate=no>_^_1_^_</span> \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0 \u0dc0\u0dda </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0dc4\u0dd2\u0dc3\u0d9a\u0da7 \u0d87\u0dad\u0dd2 \u0dbd\u0d9a\u0dca\u0dc2\u0dab \u0d9c\u0dab\u0db1</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of tokens in the vocabulary </li>\n<li><span translate=no>_^_1_^_</span> is the number of features in embeddings </li>\n<li><span translate=no>_^_2_^_</span> is the number of layers in the decoder <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> are the layers with cross attention <span translate=no>_^_5_^_</span> </li>\n<li><span translate=no>_^_6_^_</span> is the length of a chunk </li>\n<li><span translate=no>_^_7_^_</span> is the number of heads in attention layers </li>\n<li><span translate=no>_^_8_^_</span> is the size of attention heads </li>\n<li><span translate=no>_^_9_^_</span> is the size of the feed-forward networks hidden layers </li>\n<li><span translate=no>_^_10_^_</span> is the nearest neighbor encoder</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0\u0dda \u0da7\u0ddd\u0d9a\u0db1 \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_1_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dbd \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba\u0dda \u0dc3\u0dca\u0dae\u0dbb \u0d9c\u0dab\u0db1 <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> \u0dc4\u0dbb\u0dc3\u0dca \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0d87\u0dad\u0dd2 \u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dda <span translate=no>_^_5_^_</span> </li>\n<li><span translate=no>_^_6_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a \u0daf\u0dd2\u0d9c </li>\n<li><span translate=no>_^_7_^_</span> \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dbd \u0dc4\u0dd2\u0dc3\u0dca \u0d9c\u0dab\u0db1 </li>\n<li><span translate=no>_^_8_^_</span> \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1\u0dd3\u0db1\u0dca \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </li>\n<li><span translate=no>_^_9_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0ddd\u0dc2\u0d9a \u0d89\u0daf\u0dd2\u0dbb\u0dd2 \u0da2\u0dcf\u0dbd\u0dba\u0dda \u0dc3\u0dd0\u0d9f\u0dc0\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb \u0dc0\u0dbd \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dba\u0dd2 </li>\n<li><span translate=no>_^_10_^_</span> \u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba\u0dba\u0dd2</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer embeddings of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0dba\u0db1\u0dd4 \u0dc4\u0dd0\u0da9\u0dba\u0dda \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0dc0\u0dda <span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"RETRO model": "\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba",
|
||||
"RETRO model with encoder for neighbors and autoregressive decoder": "\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba \u0dc3\u0dc4 \u0dc3\u0dca\u0dc0\u0dba\u0d82\u0d9a\u0dca\u0dbb\u0dd3\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0d9c\u0dcf\u0db8\u0dd3 \u0dc0\u0dd2\u0d9a\u0dda\u0dad\u0d9a\u0dba \u0dc3\u0dc4\u0dd2\u0dad \u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
{
|
||||
"<h1>RETRO model</h1>\n<p>This is the model definition for <a href=\"index.html\">RETRO</a>.</p>\n": "<h1>\u590d\u53e4\u6a21\u578b</h1>\n<p>\u8fd9\u662f RETRO \u7684\u6a21\u578b<a href=\"index.html\">\u5b9a\u4e49</a>\u3002</p>\n",
|
||||
"<h2><a href=\"../rope/index.html\">RoPE embeddings</a></h2>\n<p><em>We use rotary position embeddings in self-attention layers. We assume the positional information gets embedded in embeddings and therefore not use them in causal attention. <a href=\"https://arxiv.org/abs/3999902edc8511eba3db37f65e372566\">Non-causal self-attention needs explicit positional information because it cannot infer it</a>.</em></p>\n": "<h2><a href=\"../rope/index.html\">\u7ef3\u7d22\u5d4c\u5165</a></h2>\n<p><em>\u6211\u4eec\u5728\u81ea\u6211\u6ce8\u610f\u529b\u5c42\u4e2d\u4f7f\u7528\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165\u3002\u6211\u4eec\u5047\u8bbe\u4f4d\u7f6e\u4fe1\u606f\u88ab\u5d4c\u5165\u5230\u5d4c\u5165\u4e2d\uff0c\u56e0\u6b64\u4e0d\u4f1a\u5728\u56e0\u679c\u5173\u6ce8\u4e2d\u4f7f\u7528\u5b83\u4eec\u3002<a href=\"https://arxiv.org/abs/3999902edc8511eba3db37f65e372566\">\u975e\u56e0\u679c\u7684\u81ea\u6211\u6ce8\u610f\u529b\u9700\u8981\u660e\u786e\u7684\u4f4d\u7f6e\u4fe1\u606f\uff0c\u56e0\u4e3a\u5b83\u65e0\u6cd5\u63a8\u65ad\u51fa\u6765</a>\u3002</em></p>\n",
|
||||
"<h2>Chunked Cross-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This is similar to the cross-attention layer defined above.</p>\n<p>This is used in the decoder to pay attention to the retrieved neighbor chunks.</p>\n<p><em>We do not use any explicit positional embeddings here. We assume that the model can represent positional information in the embeddings implicitly.</em></p>\n": "<h2>\u5206\u5757\u4ea4\u53c9\u6ce8\u610f\u529b\u5c42<span translate=no>_^_0_^_</span></h2>\n<p>\u8fd9\u4e0e\u4e0a\u9762\u5b9a\u4e49\u7684\u4ea4\u53c9\u6ce8\u610f\u529b\u5c42\u7c7b\u4f3c\u3002</p>\n<p>\u8fd9\u5728\u89e3\u7801\u5668\u4e2d\u7528\u4e8e\u5173\u6ce8\u68c0\u7d22\u5230\u7684\u90bb\u5c45\u5757\u3002</p>\n<p><em>\u6211\u4eec\u5728\u6b64\u5904\u4e0d\u4f7f\u7528\u4efb\u4f55\u663e\u5f0f\u7684\u4f4d\u7f6e\u5d4c\u5165\u3002\u6211\u4eec\u5047\u8bbe\u6a21\u578b\u53ef\u4ee5\u5728\u5d4c\u5165\u4e2d\u9690\u5f0f\u8868\u793a\u4f4d\u7f6e\u4fe1\u606f\u3002</em></p>\n",
|
||||
"<h2>Cross-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This is similar to the self-attention layer defined above, except that it gets keys and values from a different set of embeddings than the queries.</p>\n<p>This is used in the encoder to encode the retrieved chunks based on the input chunks.</p>\n<p><em>We do not use any explicit positional embeddings here. We assume that the model can represent positional information in the embeddings implicitly.</em></p>\n": "<h2>\u4ea4\u53c9\u6ce8\u610f\u529b\u5c42<span translate=no>_^_0_^_</span></h2>\n<p>\u8fd9\u4e0e\u4e0a\u9762\u5b9a\u4e49\u7684\u81ea\u6211\u6ce8\u610f\u5c42\u7c7b\u4f3c\uff0c\u4e0d\u540c\u4e4b\u5904\u5728\u4e8e\u5b83\u4ece\u4e0e\u67e5\u8be2\u4e0d\u540c\u7684\u5d4c\u5165\u96c6\u83b7\u53d6\u952e\u548c\u503c\u3002</p>\n<p>\u8fd9\u5728\u7f16\u7801\u5668\u4e2d\u7528\u4e8e\u6839\u636e\u8f93\u5165\u533a\u5757\u5bf9\u68c0\u7d22\u5230\u7684\u533a\u5757\u8fdb\u884c\u7f16\u7801\u3002</p>\n<p><em>\u6211\u4eec\u5728\u6b64\u5904\u4e0d\u4f7f\u7528\u4efb\u4f55\u663e\u5f0f\u7684\u4f4d\u7f6e\u5d4c\u5165\u3002\u6211\u4eec\u5047\u8bbe\u6a21\u578b\u53ef\u4ee5\u5728\u5d4c\u5165\u4e2d\u9690\u5f0f\u8868\u793a\u4f4d\u7f6e\u4fe1\u606f\u3002</em></p>\n",
|
||||
"<h2>Nearest Neighbor Encoder <span translate=no>_^_0_^_</span></h2>\n<p>This module encodes the retrieved nearest neighbors</p>\n": "<h2>\u6700\u8fd1\u90bb\u7f16\u7801\u5668<span translate=no>_^_0_^_</span></h2>\n<p>\u6b64\u6a21\u5757\u5bf9\u68c0\u7d22\u5230\u7684\u6700\u8fd1\u90bb\u8fdb\u884c\u7f16\u7801</p>\n",
|
||||
"<h2>Retro Model</h2>\n<p>This is the Retro decoder</p>\n": "<h2>\u590d\u53e4\u6a21\u7279</h2>\n<p>\u8fd9\u662f\u590d\u53e4\u89e3\u7801\u5668</p>\n",
|
||||
"<h2>Self-Attention Layer <span translate=no>_^_0_^_</span></h2>\n<p>This applies causal and non-causal <a href=\"../mha.html\">multi-headed self-attention</a>.</p>\n": "<h2>\u81ea\u6211\u6ce8\u610f\u5c42<span translate=no>_^_0_^_</span></h2>\n<p>\u8fd9\u9002\u7528\u4e8e\u56e0\u679c\u548c\u975e\u56e0\u679c\u7684<a href=\"../mha.html\">\u591a\u5934\u81ea\u6211\u5173\u6ce8</a>\u3002</p>\n",
|
||||
"<h3>Mask the attention layer for causal attention</h3>\n<ul><li><span translate=no>_^_0_^_</span> is the attention matrix of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<h3>\u906e\u4f4f\u6ce8\u610f\u5c42\u4ee5\u83b7\u5f97\u56e0\u679c\u5173\u6ce8</h3>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u7684\u6ce8\u610f\u529b\u77e9\u9635<span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<h3>Position-wise Feed Forward Layer <span translate=no>_^_0_^_</span></h3>\n<p>This consists of two linear layers and an activation in the middle.</p>\n": "<h3>\u4f4d\u7f6e\u524d\u9988\u5c42<span translate=no>_^_0_^_</span></h3>\n<p>\u5b83\u7531\u4e24\u4e2a\u7ebf\u6027\u5c42\u548c\u4e2d\u95f4\u7684\u6fc0\u6d3b\u5c42\u7ec4\u6210\u3002</p>\n",
|
||||
"<h3>Test the model with fake data</h3>\n": "<h3>\u4f7f\u7528\u865a\u5047\u6570\u636e\u6d4b\u8bd5\u6a21\u578b</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> are the embeddings of shape <span translate=no>_^_1_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6\u7684\u5d4c\u5165<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p> <span translate=no>_^_0_^_</span> are the input embeddings of shape <span translate=no>_^_1_^_</span> <span translate=no>_^_2_^_</span> are the retrieved nearest neighbors of shape <span translate=no>_^_3_^_</span></p>\n": "<p><span translate=no>_^_0_^_</span>shape \u7684\u8f93\u5165\u5d4c\u5165<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u662f\u68c0\u7d22\u5230\u7684 shape \u7684\u6700\u8fd1\u90bb\u503c<span translate=no>_^_3_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span></p>\n<p>We passed the embeddings of <span translate=no>_^_1_^_</span> to encoder. </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n<p>\u6211\u4eec\u5c06\u7684\u5d4c\u5165\u4f20\u9012<span translate=no>_^_1_^_</span>\u7ed9\u7f16\u7801\u5668\u3002</p>\n",
|
||||
"<p>Activation </p>\n": "<p>\u6fc0\u6d3b</p>\n",
|
||||
"<p>Add residual connection </p>\n": "<p>\u6dfb\u52a0\u5269\u4f59\u8fde\u63a5</p>\n",
|
||||
"<p>Add the residual connection </p>\n": "<p>\u6dfb\u52a0\u5269\u4f59\u8fde\u63a5</p>\n",
|
||||
"<p>Append <span translate=no>_^_0_^_</span> zero embedding to the left; i.e. right shift it back </p>\n": "<p>\u5411\u5de6\u8ffd\u52a0<span translate=no>_^_0_^_</span>\u96f6\u5d4c\u5165\uff1b\u5373\u53f3\u79fb\u56de\u53bb</p>\n",
|
||||
"<p>Append empty embeddings to the end to be able to split the input into chunks </p>\n": "<p>\u5728\u672b\u5c3e\u8ffd\u52a0\u7a7a\u5d4c\u5165\uff0c\u4ee5\u4fbf\u80fd\u591f\u5c06\u8f93\u5165\u62c6\u5206\u4e3a\u5757</p>\n",
|
||||
"<p>Apply final linear layer. The result will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5e94\u7528\u6700\u540e\u7684\u7ebf\u6027\u56fe\u5c42\u3002\u7ed3\u679c\u5c06\u6709\u5f62\u72b6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Apply masks if it's causal attention </p>\n": "<p>\u5982\u679c\u662f\u56e0\u679c\u5173\u7cfb\uff0c\u8bf7\u6234\u53e3\u7f69</p>\n",
|
||||
"<p>Apply rotary positional embeddings </p>\n": "<p>\u5e94\u7528\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165</p>\n",
|
||||
"<p>Apply softmax over the last two dimensions <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5728\u6700\u540e\u4e24\u4e2a\u7ef4\u5ea6\u4e0a\u5e94\u7528 softmax<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Attention layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6ce8\u610f\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Bi-directional self attention <span translate=no>_^_0_^_</span> </p>\n": "<p>\u53cc\u5411\u81ea\u6211\u5173\u6ce8<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Bi-directional self attention layers </p>\n": "<p>\u53cc\u5411\u81ea\u6211\u5173\u6ce8\u5c42</p>\n",
|
||||
"<p>Calculate <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate attention probabilities </p>\n": "<p>\u8ba1\u7b97\u6ce8\u610f\u529b\u6982\u7387</p>\n",
|
||||
"<p>Calculate attention scores for all chunks. Each retrieved neighbor will pay attention to the original chunk that retrieved it. This will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u6240\u6709\u533a\u5757\u7684\u6ce8\u610f\u529b\u5206\u6570\u3002\u6bcf\u4e2a\u68c0\u7d22\u5230\u7684\u90bb\u5c45\u90fd\u5c06\u6ce8\u610f\u68c0\u7d22\u5230\u5b83\u7684\u539f\u59cb\u533a\u5757\u3002\u8fd9\u5c06\u6709\u5f62\u72b6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate attention scores for input chunks. Each chunk will pay attention to neighbors retrieved by the previous chunk. This will have shape <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u8f93\u5165\u533a\u5757\u7684\u6ce8\u610f\u529b\u5206\u6570\u3002\u6bcf\u4e2a\u533a\u5757\u90fd\u5c06\u5173\u6ce8\u524d\u4e00\u4e2a\u533a\u5757\u68c0\u7d22\u5230\u7684\u90bb\u5c45\u3002\u8fd9\u5c06\u6709\u5f62\u72b6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate attentions </p>\n": "<p>\u8ba1\u7b97\u6ce8\u610f\u529b</p>\n",
|
||||
"<p>Calculate softmax across the last dimension </p>\n": "<p>\u8ba1\u7b97\u6700\u540e\u4e00\u4e2a\u7ef4\u5ea6\u7684 softmax</p>\n",
|
||||
"<p>Calculate the product of position index and <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u6301\u4ed3\u6307\u6570\u7684\u4e58\u79ef\u548c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate</p>\n<span translate=no>_^_0_^_</span><p>for <span translate=no>_^_1_^_</span> </p>\n": "<p>\u8ba1\u7b97</p>\n<span translate=no>_^_0_^_</span><p>\u5bf9\u4e8e<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Causal self attention <span translate=no>_^_0_^_</span> </p>\n": "<p>\u56e0\u679c\u81ea\u6211\u5173\u6ce8<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Change from shape <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u4ece\u5f62\u72b6\u6539<span translate=no>_^_0_^_</span>\u4e3a<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Chunked cross attention layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5206\u5757\u4ea4\u53c9\u6ce8\u610f\u529b\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Chunked-cross attention if <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5927\u5757\u4ea4\u53c9\u6ce8\u610f\u5982\u679c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Concatenate so that for row <span translate=no>_^_0_^_</span> we have <span translate=no>_^_1_^_</span> </p>\n": "<p>\u8fde\u63a5\u8fd9\u6837<span translate=no>_^_0_^_</span>\u6211\u4eec\u5c31\u6709 row<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Create a triangular mask </p>\n": "<p>\u521b\u5efa\u4e09\u89d2\u5f62\u8499\u7248</p>\n",
|
||||
"<p>Create position indexes <span translate=no>_^_0_^_</span> </p>\n": "<p>\u521b\u5efa\u5934\u5bf8\u6307\u6570<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Cross attention if <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4ea4\u53c9\u6ce8\u610f\u5982\u679c<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Cross-attention layers </p>\n": "<p>\u4ea4\u53c9\u6ce8\u610f\u5c42</p>\n",
|
||||
"<p>Embeddings of the retrieved neighbors <span translate=no>_^_0_^_</span>.</p>\n<p>We use same embeddings for both input and neighbors </p>\n": "<p>\u68c0\u7d22\u5230\u7684\u90bb\u5c45\u7684\u5d4c\u5165<span translate=no>_^_0_^_</span>\u3002</p>\n<p>\u6211\u4eec\u5bf9\u8f93\u5165\u548c\u90bb\u5c45\u4f7f\u7528\u76f8\u540c\u7684\u5d4c\u5165</p>\n",
|
||||
"<p>Extract the shape </p>\n": "<p>\u63d0\u53d6\u5f62\u72b6</p>\n",
|
||||
"<p>Feed forward layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u524d\u9988\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Feed forward layers </p>\n": "<p>\u524d\u9988\u56fe\u5c42</p>\n",
|
||||
"<p>Feed forward layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u524d\u9988\u56fe\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Filter by the mask </p>\n": "<p>\u6309\u53e3\u7f69\u8fc7\u6ee4</p>\n",
|
||||
"<p>Final linear layer </p>\n": "<p>\u6700\u540e\u7684\u7ebf\u6027\u5c42</p>\n",
|
||||
"<p>First linear layer </p>\n": "<p>\u7b2c\u4e00\u4e2a\u7ebf\u6027\u5c42</p>\n",
|
||||
"<p>For all layers <span translate=no>_^_0_^_</span> </p>\n": "<p>\u9002\u7528\u4e8e\u6240\u6709\u56fe\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Gather values </p>\n": "<p>\u6536\u96c6\u503c</p>\n",
|
||||
"<p>Get encoder embeddings before the first <span translate=no>_^_0_^_</span> layer, when <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5728\u7b2c\u4e00<span translate=no>_^_0_^_</span>\u5c42\u4e4b\u524d\u83b7\u53d6\u7f16\u7801\u5668\u5d4c\u5165<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get input embeddings <span translate=no>_^_0_^_</span> </p>\n": "<p>\u83b7\u53d6\u8f93\u5165\u5d4c\u5165<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get keys and values from the input chunks </p>\n": "<p>\u4ece\u8f93\u5165\u5757\u4e2d\u83b7\u53d6\u952e\u548c\u503c</p>\n",
|
||||
"<p>Get keys and values from the retrieved neighbors </p>\n": "<p>\u4ece\u68c0\u7d22\u5230\u7684\u90bb\u5c45\u83b7\u53d6\u952e\u548c\u503c</p>\n",
|
||||
"<p>Get query from the input </p>\n": "<p>\u4ece\u8f93\u5165\u4e2d\u83b7\u53d6\u67e5\u8be2</p>\n",
|
||||
"<p>Get query from the retrieved chunks </p>\n": "<p>\u4ece\u68c0\u7d22\u5230\u7684\u533a\u5757\u4e2d\u83b7\u53d6\u67e5\u8be2</p>\n",
|
||||
"<p>Get query, key, and values and split them in to heads. These will have shapes <span translate=no>_^_0_^_</span> </p>\n": "<p>\u83b7\u53d6\u67e5\u8be2\u3001\u952e\u548c\u503c\uff0c\u5e76\u5c06\u5b83\u4eec\u5206\u6210\u5934\u90e8\u3002\u8fd9\u4e9b\u4f1a\u6709\u5f62\u72b6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Get shape </p>\n": "<p>\u5851\u9020\u8eab\u6750</p>\n",
|
||||
"<p>Get values </p>\n": "<p>\u83b7\u53d6\u503c</p>\n",
|
||||
"<p>Increment chunked cross-attention index </p>\n": "<p>\u9012\u589e\u5206\u5757\u4ea4\u53c9\u6ce8\u610f\u529b\u6307\u6570</p>\n",
|
||||
"<p>Incremnt the cross attention index </p>\n": "<p>\u589e\u52a0\u4ea4\u53c9\u6ce8\u610f\u529b\u6307\u6570</p>\n",
|
||||
"<p>Keep index of the chunked cross attention layer </p>\n": "<p>\u4fdd\u7559\u5206\u5757\u4ea4\u53c9\u6ce8\u610f\u5c42\u7684\u7d22\u5f15</p>\n",
|
||||
"<p>Keep the index of the cross attention layer </p>\n": "<p>\u4fdd\u7559\u4ea4\u53c9\u5173\u6ce8\u5c42\u7684\u7d22\u5f15</p>\n",
|
||||
"<p>Linear layers for query, key and value heads. </p>\n": "<p>\u7528\u4e8e\u67e5\u8be2\u3001\u952e\u548c\u503c\u6807\u5934\u7684\u7ebf\u6027\u56fe\u5c42\u3002</p>\n",
|
||||
"<p>No attention if there are no chunks (for short inputs when sampling) </p>\n": "<p>\u5982\u679c\u6ca1\u6709\u533a\u5757\uff0c\u5219\u4e0d\u6ce8\u610f\uff08\u91c7\u6837\u65f6\u7528\u4e8e\u77ed\u8f93\u5165\uff09</p>\n",
|
||||
"<p>No masking for non-causal attention </p>\n": "<p>\u975e\u56e0\u679c\u6ce8\u610f\u6ca1\u6709\u906e\u7f69</p>\n",
|
||||
"<p>Normalize encoder embeddings </p>\n": "<p>\u89c4\u8303\u5316\u7f16\u7801\u5668\u5d4c\u5165</p>\n",
|
||||
"<p>Normalize retrieved chunks </p>\n": "<p>\u89c4\u8303\u5316\u68c0\u7d22\u5230\u7684\u533a\u5757</p>\n",
|
||||
"<p>Pre-norm </p>\n": "<p>\u89c4\u8303\u524d</p>\n",
|
||||
"<p>Pre-norm layer </p>\n": "<p>\u89c4\u8303\u524d\u5c42</p>\n",
|
||||
"<p>Pre-norm layer for the query embeddings. The paper uses RMSNorm instead. </p>\n": "<p>\u67e5\u8be2\u5d4c\u5165\u7684\u9884\u89c4\u8303\u5c42\u3002\u672c\u6587\u6539\u4e3a\u4f7f\u7528 rmsNorm\u3002</p>\n",
|
||||
"<p>Pre-norm layer. The paper uses RMSNorm instead. </p>\n": "<p>\u9884\u5148\u89c4\u8303\u5c42\u3002\u672c\u6587\u6539\u4e3a\u4f7f\u7528 rmsNorm\u3002</p>\n",
|
||||
"<p>Pre-normalization </p>\n": "<p>\u89c4\u8303\u5316\u524d</p>\n",
|
||||
"<p>Pre-normalization layer for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u9884\u5f52\u4e00\u5316\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Pre-normalization layer for nearest neighbor embeddings from <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6700\u8fd1\u90bb\u5d4c\u5165\u7684\u9884\u5f52\u4e00\u5316\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>ReLU Activation </p>\n": "<p>ReLU \u6fc0\u6d3b</p>\n",
|
||||
"<p>Readout layer <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8bfb\u51fa\u5c42<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Remove the first <span translate=no>_^_0_^_</span> embeddings. The input pays attention to neighbors retrieved and encoded using the past tokens only; so that there is no information leakage. That is the retrieved neighbors from the first chunks will have information from the first chunk. So by shifting the sequence to the left by <span translate=no>_^_1_^_</span> we make sure that information only flows to the right. </p>\n": "<p>\u79fb\u9664\u7b2c\u4e00\u4e2a<span translate=no>_^_0_^_</span>\u5d4c\u5165\u3002\u8f93\u5165\u53ea\u5173\u6ce8\u4f7f\u7528\u8fc7\u53bb\u7684\u4ee4\u724c\u68c0\u7d22\u548c\u7f16\u7801\u7684\u90bb\u5c45\uff1b\u8fd9\u6837\u5c31\u4e0d\u4f1a\u6cc4\u9732\u4fe1\u606f\u3002\u4e5f\u5c31\u662f\u8bf4\uff0c\u4ece\u7b2c\u4e00\u4e2a\u533a\u5757\u4e2d\u68c0\u7d22\u5230\u7684\u90bb\u5c45\u5c06\u83b7\u5f97\u6765\u81ea\u7b2c\u4e00\u4e2a\u533a\u5757\u7684\u4fe1\u606f\u3002\u56e0\u6b64\uff0c\u901a\u8fc7\u5c06\u5e8f\u5217\u5411\u5de6\u79fb\u52a8\uff0c<span translate=no>_^_1_^_</span>\u6211\u4eec\u53ef\u4ee5\u786e\u4fdd\u4fe1\u606f\u53ea\u5411\u53f3\u6d41\u52a8\u3002</p>\n",
|
||||
"<p>Reshape the input into chunks. </p>\n": "<p>\u5c06\u8f93\u5165\u91cd\u5851\u4e3a\u5757\u3002</p>\n",
|
||||
"<p>Residual </p>\n": "<p>\u5269\u4f59</p>\n",
|
||||
"<p>Residual connection </p>\n": "<p>\u5269\u4f59\u8fde\u63a5</p>\n",
|
||||
"<p>Rotary positional embeddings </p>\n": "<p>\u65cb\u8f6c\u4f4d\u7f6e\u5d4c\u5165</p>\n",
|
||||
"<p>Scale attention scores </p>\n": "<p>\u7f29\u653e\u6ce8\u610f\u529b\u5206\u6570</p>\n",
|
||||
"<p>Scale it by <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6309\u6bd4\u4f8b\u7f29\u653e<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Second linear layer </p>\n": "<p>\u7b2c\u4e8c\u4e2a\u7ebf\u6027\u5c42</p>\n",
|
||||
"<p>Softmax for attention probabilities </p>\n": "<p>Softmax \u8868\u793a\u6ce8\u610f\u529b\u6982\u7387</p>\n",
|
||||
"<p>The two linear layers </p>\n": "<p>\u4e24\u4e2a\u7ebf\u6027\u5c42</p>\n",
|
||||
"<p>To scale attentions before softmax by <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5728 softmax \u4e4b\u524d\u6269\u5927\u6ce8\u610f\u529b<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Token embedding layer </p>\n": "<p>\u4ee4\u724c\u5d4c\u5165\u5c42</p>\n",
|
||||
"<p>Truncate and add the residual connection </p>\n": "<p>\u622a\u65ad\u5e76\u6dfb\u52a0\u5269\u4f59\u8fde\u63a5</p>\n",
|
||||
"<p>return <span translate=no>_^_0_^_</span> </p>\n": "<p>\u8fd4\u56de<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are the retrieved nearest neighbor chunk embeddings with shape <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> are the input chunks from which the nearest neighbors were retrieved with shape <span translate=no>_^_3_^_</span>. This is already normalized.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5e26\u6709 shape \u7684\u68c0\u7d22\u7684\u6700\u8fd1\u90bb\u533a\u5757\u5d4c\u5165<span translate=no>_^_1_^_</span></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u4f7f\u7528 shape \u4ece\u4e2d\u68c0\u7d22\u6700\u8fd1\u90bb\u57df\u7684\u8f93\u5165\u5757<span translate=no>_^_3_^_</span>\u3002\u8fd9\u5df2\u7ecf\u89c4\u8303\u5316\u4e86\u3002</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> are token embeddings of the retrieved nearest neighbors, <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span></li></ul>\n<ul><li><span translate=no>_^_3_^_</span> is are the input token embeddings, <span translate=no>_^_4_^_</span> of shape <span translate=no>_^_5_^_</span></li></ul>\n<p><em>The chunks <span translate=no>_^_6_^_</span> and neighbors <span translate=no>_^_7_^_</span> are processed in parallel.</em></p>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u68c0\u7d22\u5230\u7684\u6700\u8fd1\u90bb\u7684\u4ee4\u724c\u5d4c\u5165\uff0c\u5f62<span translate=no>_^_1_^_</span>\u72b6\u4e3a<span translate=no>_^_2_^_</span></li></ul>\n<ul><li><span translate=no>_^_3_^_</span>is \u662f\u5f62\u72b6\u7684\u8f93\u5165\u4ee4\u724c\u5d4c\u5165<span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span></li></ul>\n<p><em>\u533a\u5757<span translate=no>_^_6_^_</span>\u548c\u90bb\u5c45<span translate=no>_^_7_^_</span>\u662f\u5e76\u884c\u5904\u7406\u7684\u3002</em></p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the Tensor at the head of a key or a query with shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u4f4d\u4e8e\u952e\u6216\u5e26\u6709\u5f62\u72b6\u7684\u67e5\u8be2\u5f00\u5934\u7684 Tensor<span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the input sequence, <span translate=no>_^_1_^_</span> of shape <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> are the retrieved neighbors <span translate=no>_^_4_^_</span> of shape <span translate=no>_^_5_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u5f62\u72b6<span translate=no>_^_1_^_</span>\u7684\u8f93\u5165\u5e8f\u5217<span translate=no>_^_2_^_</span></li>\n</ul><li><span translate=no>_^_3_^_</span>\u662f\u68c0\u7d22\u5230<span translate=no>_^_4_^_</span>\u7684\u5f62\u72b6\u90bb\u57df<span translate=no>_^_5_^_</span></li>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the length of a chunk </li>\n<li><span translate=no>_^_1_^_</span> is the number of layers in the encoder <span translate=no>_^_2_^_</span> </li>\n<li><span translate=no>_^_3_^_</span> are the layers with cross attention <span translate=no>_^_4_^_</span> </li>\n<li><span translate=no>_^_5_^_</span> is the number of features in embeddings </li>\n<li><span translate=no>_^_6_^_</span> is the number of heads in attention layers </li>\n<li><span translate=no>_^_7_^_</span> is the size of attention heads </li>\n<li><span translate=no>_^_8_^_</span> is the size of the feed-forward networks hidden layers</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u533a\u5757\u7684\u957f\u5ea6</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u7f16\u7801\u5668\u4e2d\u7684\u5c42\u6570<span translate=no>_^_2_^_</span></li>\n<li><span translate=no>_^_3_^_</span>\u662f\u4ea4\u53c9\u5173\u6ce8\u7684\u5c42\u6b21\u5417<span translate=no>_^_4_^_</span></li>\n<li><span translate=no>_^_5_^_</span>\u662f\u5d4c\u5165\u4e2d\u8981\u7d20\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_6_^_</span>\u662f\u6ce8\u610f\u5c42\u4e2d\u7684\u5934\u90e8\u6570\u91cf</li>\n<li><span translate=no>_^_7_^_</span>\u662f\u6ce8\u610f\u5934\u7684\u5927\u5c0f</li>\n<li><span translate=no>_^_8_^_</span>\u662f\u524d\u9988\u7f51\u7edc\u9690\u85cf\u5c42\u7684\u5927\u5c0f</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features <span translate=no>_^_1_^_</span> </li>\n<li><span translate=no>_^_2_^_</span> is the constant used for calculating <span translate=no>_^_3_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8981\u7d20\u7684\u6570\u91cf<span translate=no>_^_1_^_</span></li>\n</ul><li><span translate=no>_^_2_^_</span>\u662f\u7528\u4e8e\u8ba1\u7b97\u7684\u5e38\u6570<span translate=no>_^_3_^_</span></li>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number features in the hidden layer</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668\u5d4c\u5165\u4e2d\u7684\u7279\u5f81\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u9690\u85cf\u56fe\u5c42\u4e2d\u7684\u6570\u5b57\u8981\u7d20</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head </li>\n<li><span translate=no>_^_3_^_</span> indicates whether this is causal attention (masked)</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668\u5d4c\u5165\u4e2d\u7684\u7279\u5f81\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u6ce8\u610f\u5934\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u6bcf\u5934\u7279\u5f81\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_3_^_</span>\u8868\u793a\u8fd9\u662f\u5426\u662f\u56e0\u679c\u5173\u6ce8\uff08\u5c4f\u853d\uff09</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head </li>\n<li><span translate=no>_^_3_^_</span> is the length of a chunk</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668\u5d4c\u5165\u4e2d\u7684\u7279\u5f81\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u6ce8\u610f\u5934\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u6bcf\u5934\u7279\u5f81\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_3_^_</span>\u662f\u533a\u5757\u7684\u957f\u5ea6</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of features in transformer embeddings </li>\n<li><span translate=no>_^_1_^_</span> is the number of attention heads </li>\n<li><span translate=no>_^_2_^_</span> is the number of features per head</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u53d8\u538b\u5668\u5d4c\u5165\u4e2d\u7684\u7279\u5f81\u6570</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u6ce8\u610f\u5934\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u6bcf\u5934\u7279\u5f81\u7684\u6570\u91cf</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the number of tokens in the vocabulary </li>\n<li><span translate=no>_^_1_^_</span> is the number of features in embeddings </li>\n<li><span translate=no>_^_2_^_</span> is the number of layers in the decoder <span translate=no>_^_3_^_</span> </li>\n<li><span translate=no>_^_4_^_</span> are the layers with cross attention <span translate=no>_^_5_^_</span> </li>\n<li><span translate=no>_^_6_^_</span> is the length of a chunk </li>\n<li><span translate=no>_^_7_^_</span> is the number of heads in attention layers </li>\n<li><span translate=no>_^_8_^_</span> is the size of attention heads </li>\n<li><span translate=no>_^_9_^_</span> is the size of the feed-forward networks hidden layers </li>\n<li><span translate=no>_^_10_^_</span> is the nearest neighbor encoder</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8bcd\u6c47\u8868\u4e2d\u4ee3\u5e01\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5d4c\u5165\u4e2d\u8981\u7d20\u7684\u6570\u91cf</li>\n<li><span translate=no>_^_2_^_</span>\u662f\u89e3\u7801\u5668\u4e2d\u7684\u5c42\u6570<span translate=no>_^_3_^_</span></li>\n<li><span translate=no>_^_4_^_</span>\u662f\u4ea4\u53c9\u5173\u6ce8\u7684\u5c42\u6b21\u5417<span translate=no>_^_5_^_</span></li>\n<li><span translate=no>_^_6_^_</span>\u662f\u533a\u5757\u7684\u957f\u5ea6</li>\n<li><span translate=no>_^_7_^_</span>\u662f\u6ce8\u610f\u5c42\u4e2d\u7684\u5934\u90e8\u6570\u91cf</li>\n<li><span translate=no>_^_8_^_</span>\u662f\u6ce8\u610f\u5934\u7684\u5927\u5c0f</li>\n<li><span translate=no>_^_9_^_</span>\u662f\u524d\u9988\u7f51\u7edc\u9690\u85cf\u5c42\u7684\u5927\u5c0f</li>\n<li><span translate=no>_^_10_^_</span>\u662f\u6700\u8fd1\u90bb\u7f16\u7801\u5668</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the transformer embeddings of shape <span translate=no>_^_1_^_</span></li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u53d8\u538b\u5668\u5d4c\u5165\u7684\u5f62\u72b6\u662f\u591a\u5c11<span translate=no>_^_1_^_</span></li></ul>\n",
|
||||
"RETRO model": "\u590d\u53e4\u578b\u53f7",
|
||||
"RETRO model with encoder for neighbors and autoregressive decoder": "\u5e26\u6709\u90bb\u5c45\u7f16\u7801\u5668\u548c\u81ea\u56de\u5f52\u89e3\u7801\u5668\u7684RETRO\u6a21\u578b"
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"<h1>RETRO training</h1>\n<p>This is the training code for <a href=\"index.html\">RETRO</a>.</p>\n": "<h1>\u30ec\u30c8\u30ed\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</h1>\n<p><a href=\"index.html\">\u3053\u308c\u306fRETRO\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u3067\u3059</a>\u3002</p>\n",
|
||||
"<h2>Create and train a small model</h2>\n": "<h2>\u5c0f\u578b\u30e2\u30c7\u30eb\u306e\u4f5c\u6210\u3068\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</h2>\n",
|
||||
"<h2>Retro trainer</h2>\n": "<h2>\u30ec\u30c8\u30ed\u30c8\u30ec\u30fc\u30ca\u30fc</h2>\n",
|
||||
"<h2>Sampler</h2>\n<p>This class greedily samples from a model.</p>\n": "<h2>\u30b5\u30f3\u30d7\u30e9\u30fc</h2>\n<p>\u3053\u306e\u30af\u30e9\u30b9\u306f\u30e2\u30c7\u30eb\u304b\u3089\u8caa\u6b32\u306b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<h3>Retrieve nearest neighbors of a given chunk</h3>\n": "<h3>\u4e0e\u3048\u3089\u308c\u305f\u30c1\u30e3\u30f3\u30af\u306e\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u306e\u30c7\u30fc\u30bf\u3092\u53d6\u5f97</h3>\n",
|
||||
"<h3>Sample text from the given prompt</h3>\n": "<h3>\u4e0e\u3048\u3089\u308c\u305f\u30d7\u30ed\u30f3\u30d7\u30c8\u306e\u30b5\u30f3\u30d7\u30eb\u30c6\u30ad\u30b9\u30c8</h3>\n",
|
||||
"<h3>Train the model for an epoch</h3>\n": "<h3>\u4e00\u6642\u4ee3\u3092\u5148\u53d6\u308a\u3057\u305f\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><a href=\"database.html\">Retro index</a> </p>\n": "<p><a href=\"database.html\">\u30ec\u30c8\u30ed\u30a4\u30f3\u30c7\u30c3\u30af\u30b9</a></p>\n",
|
||||
"<p>Add the sampled token text to the prompt and sample text </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u305f\u30c8\u30fc\u30af\u30f3\u306e\u30c6\u30ad\u30b9\u30c8\u3092\u30d7\u30ed\u30f3\u30d7\u30c8\u3068\u30b5\u30f3\u30d7\u30eb\u30c6\u30ad\u30b9\u30c8\u306b\u8ffd\u52a0\u3057\u307e\u3059</p>\n",
|
||||
"<p>Backward pass </p>\n": "<p>\u30d0\u30c3\u30af\u30ef\u30fc\u30c9\u30d1\u30b9</p>\n",
|
||||
"<p>Calculate loss </p>\n": "<p>\u640d\u5931\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30af\u30ea\u30a2</p>\n",
|
||||
"<p>Create an experiment </p>\n": "<p>\u30c6\u30b9\u30c8\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create dataloader </p>\n": "<p>\u30c7\u30fc\u30bf\u30ed\u30fc\u30c0\u30fc\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f5c\u6210 <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create the model </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Create the nearest neighbor encoder </p>\n": "<p>\u6700\u8fd1\u508d\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create the optimizer </p>\n": "<p>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Forward pass </p>\n": "<p>\u30d5\u30a9\u30ef\u30fc\u30c9\u30d1\u30b9</p>\n",
|
||||
"<p>GPU device </p>\n": "<p>GPU \u30c7\u30d0\u30a4\u30b9</p>\n",
|
||||
"<p>Get model output </p>\n": "<p>\u30e2\u30c7\u30eb\u51fa\u529b\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the last chunk for which we haven't retrieved neighbors </p>\n": "<p>\u96a3\u63a5\u30c7\u30fc\u30bf\u3092\u53d6\u5f97\u3057\u3066\u3044\u306a\u3044\u6700\u5f8c\u306e\u30c1\u30e3\u30f3\u30af\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Get the neighbors (with neighbor length equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u8fd1\u508d\u3092\u53d6\u5f97 (\u8fd1\u508d\u306e\u9577\u3055\u304c\u3068\u7b49\u3057\u3044) <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Greedily sample the last token </p>\n": "<p>\u6700\u5f8c\u306e\u30c8\u30fc\u30af\u30f3\u3092\u6b32\u5f35\u3063\u3066\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3059\u308b</p>\n",
|
||||
"<p>Hyper-parameters </p>\n": "<p>\u30cf\u30a4\u30d1\u30fc\u30d1\u30e9\u30e1\u30fc\u30bf</p>\n",
|
||||
"<p>Iterate through training data </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30c7\u30fc\u30bf\u3092\u7e70\u308a\u8fd4\u3057\u51e6\u7406</p>\n",
|
||||
"<p>Load <a href=\"dataset.html\">Retro dataset</a> </p>\n": "<p><a href=\"dataset.html\">\u30ec\u30c8\u30ed\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u8aad\u307f\u8fbc\u307f</a></p>\n",
|
||||
"<p>Load Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u8aad\u307f\u8fbc\u307f</p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u30c7\u30fc\u30bf\u3092\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Move the model to the device </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5</p>\n",
|
||||
"<p>Move them to the same device as the model </p>\n": "<p>\u30e2\u30c7\u30eb\u3068\u540c\u3058\u30c7\u30d0\u30a4\u30b9\u306b\u79fb\u52d5\u3057\u307e\u3059</p>\n",
|
||||
"<p>Optimize the model </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u6700\u9069\u5316</p>\n",
|
||||
"<p>Print a new line </p>\n": "<p>\u65b0\u3057\u3044\u884c\u3092\u5370\u5237</p>\n",
|
||||
"<p>Retrieve nearest neighbors </p>\n": "<p>\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u3092\u691c\u7d22\u3059\u308b</p>\n",
|
||||
"<p>Retrieve the offsets of the nearest neighbors </p>\n": "<p>\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u306e\u30aa\u30d5\u30bb\u30c3\u30c8\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Sample <span translate=no>_^_0_^_</span> tokens </p>\n": "<p><span translate=no>_^_0_^_</span>\u30b5\u30f3\u30d7\u30eb\u30c8\u30fc\u30af\u30f3</p>\n",
|
||||
"<p>Sample from the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u304b\u3089\u306e\u30b5\u30f3\u30d7\u30eb <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Sampled text </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u30c6\u30ad\u30b9\u30c8</p>\n",
|
||||
"<p>Save models </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u4fdd\u5b58\u3059\u308b</p>\n",
|
||||
"<p>Save training statistics and increment the global step counter </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7d71\u8a08\u3092\u4fdd\u5b58\u3057\u3066\u30b0\u30ed\u30fc\u30d0\u30eb\u30b9\u30c6\u30c3\u30d7\u30ab\u30a6\u30f3\u30bf\u3092\u5897\u3084\u3059</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>To store nearest neighbors as strings </p>\n": "<p>\u6700\u3082\u8fd1\u3044\u8fd1\u508d\u3092\u6587\u5b57\u5217\u3068\u3057\u3066\u4fdd\u5b58\u3059\u308b\u306b\u306f</p>\n",
|
||||
"<p>Tokenize the input </p>\n": "<p>\u5165\u529b\u3092\u30c8\u30fc\u30af\u30f3\u5316</p>\n",
|
||||
"<p>Tokenize the retrieved neighbors </p>\n": "<p>\u53d6\u5f97\u3057\u305f\u30cd\u30a4\u30d0\u30fc\u3092\u30c8\u30fc\u30af\u30f3\u5316\u3059\u308b</p>\n",
|
||||
"<p>Train </p>\n": "<p>\u5217\u8eca</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
|
||||
"<p>We need to retrieve neighbors, if there are more sampled chunks than we have already retrieved for </p>\n": "<p>\u3059\u3067\u306b\u53d6\u5f97\u3057\u305f\u30c1\u30e3\u30f3\u30af\u306e\u6570\u3088\u308a\u3082\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u305f\u30c1\u30e3\u30f3\u30af\u306e\u6570\u304c\u591a\u3044\u5834\u5408\u306f\u3001\u96a3\u63a5\u30c7\u30fc\u30bf\u3092\u53d6\u5f97\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the device of the model </li>\n<li><span translate=no>_^_1_^_</span> is the <a href=\"retro.html\">Retro mode</a> </li>\n<li><span translate=no>_^_2_^_</span> is the dataloader for the <a href=\"dataset.html\">dataset with pre-retrieved neighbors</a> </li>\n<li><span translate=no>_^_3_^_</span> is the optimizer</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30e2\u30c7\u30eb\u306e\u30c7\u30d0\u30a4\u30b9\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span><a href=\"retro.html\">\u30ec\u30c8\u30ed\u30e2\u30fc\u30c9\u3067\u3059</a></li>\n<li><span translate=no>_^_2_^_</span>\u306f\u3001<a href=\"dataset.html\">\u4e8b\u524d\u306b\u8fd1\u508d\u304c\u53d6\u5f97\u3055\u308c\u305f\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306e\u30c7\u30fc\u30bf\u30ed\u30fc\u30c0\u30fc</a></li>\n<li><span translate=no>_^_3_^_</span>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3067\u3059</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the device of the model </li>\n<li><span translate=no>_^_1_^_</span> is the <a href=\"retro.html\">Retro mode</a> </li>\n<li><span translate=no>_^_2_^_</span> is the text dataset (used to get neighbor chunks) </li>\n<li><span translate=no>_^_3_^_</span> is the length of a chunk</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u30e2\u30c7\u30eb\u306e\u30c7\u30d0\u30a4\u30b9\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span><a href=\"retro.html\">\u30ec\u30c8\u30ed\u30e2\u30fc\u30c9\u3067\u3059</a></li>\n<li><span translate=no>_^_2_^_</span>\u30c6\u30ad\u30b9\u30c8\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8 (\u96a3\u63a5\u30c1\u30e3\u30f3\u30af\u306e\u53d6\u5f97\u306b\u4f7f\u7528)</li>\n<li><span translate=no>_^_3_^_</span>\u30c1\u30e3\u30f3\u30af\u306e\u9577\u3055\u3067\u3059</li></ul>\n",
|
||||
"RETRO training": "\u30ec\u30c8\u30ed\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0",
|
||||
"Training RETRO model with Tiny Shakespeare dataset": "\u30bf\u30a4\u30cb\u30fc\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u3088\u308b RETRO \u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0"
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"<h1>RETRO training</h1>\n<p>This is the training code for <a href=\"index.html\">RETRO</a>.</p>\n<p><a href=\"https://app.labml.ai/run/3113dd3ea1e711ec85ee295d18534021\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">RETRO</a>\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dda\u0dad\u0dba \u0dc0\u0dda. </p>\n<p><a href=\"https://app.labml.ai/run/3113dd3ea1e711ec85ee295d18534021\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Create and train a small model</h2>\n": "<h2>\u0d9a\u0dd4\u0da9\u0dcf\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 \u0dc3\u0dc4 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h2>\n",
|
||||
"<h2>Retro trainer</h2>\n": "<h2>\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4</h2>\n",
|
||||
"<h2>Sampler</h2>\n<p>This class greedily samples from a model.</p>\n": "<h2>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0d9a\u0dbb\u0dd4</h2>\n<p>\u0db8\u0dd9\u0db8\u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dd2\u0db1\u0dca \u0d9a\u0dd1\u0daf\u0dbb\u0d9a\u0db8\u0dd2\u0db1\u0dca \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd. </p>\n",
|
||||
"<h3>Retrieve nearest neighbors of a given chunk</h3>\n": "<h3>\u0dbd\u0db6\u0dcf\u0daf\u0dd3 \u0d87\u0dad\u0dd2 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a \u0dc5\u0d9f\u0db8 \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Sample text from the given prompt</h3>\n": "<h3>\u0dbd\u0db6\u0dcf\u0daf\u0dd3 \u0d87\u0dad\u0dd2 \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0dd9\u0db1\u0dca \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0db4\u0dd9\u0dc5</h3>\n",
|
||||
"<h3>Train the model for an epoch</h3>\n": "<h3>\u0d91\u0db4\u0ddd\u0da0\u0dca\u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p><a href=\"database.html\">Retro index</a> </p>\n": "<p><a href=\"database.html\">\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd \u0daf\u0dbb\u0dca\u0dc1\u0d9a\u0dba</a> </p>\n",
|
||||
"<p>Add the sampled token text to the prompt and sample text </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0da7\u0ddd\u0d9a\u0db1\u0dca \u0db4\u0dd9\u0dc5 \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0da7 \u0dc3\u0dc4 \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0db4\u0dd9\u0dc5\u0da7 \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Backward pass </p>\n": "<p>\u0db4\u0dc3\u0dd4\u0d9c\u0dcf\u0db8\u0dd3\u0db4\u0dcf\u0dc3\u0dca </p>\n",
|
||||
"<p>Calculate loss </p>\n": "<p>\u0d85\u0dbd\u0dcf\u0db7\u0dba\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d89\u0dc0\u0dad\u0dca </p>\n",
|
||||
"<p>Create an experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create dataloader </p>\n": "<p>\u0daf\u0dad\u0dca\u0dad\u0d9a\u0dcf\u0dbb\u0d9a\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Create the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create the nearest neighbor encoder </p>\n": "<p>\u0dc5\u0d9f\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf\u0d9c\u0dda \u0d91\u0db1\u0dca\u0d9a\u0ddd\u0da9\u0dbb\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create the optimizer </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Forward pass </p>\n": "<p>\u0d89\u0daf\u0dd2\u0dbb\u0dd2\u0dc3\u0dcf\u0db8\u0dcf\u0dbb\u0dca\u0dae\u0dba </p>\n",
|
||||
"<p>GPU device </p>\n": "<p>GPU\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba </p>\n",
|
||||
"<p>Get model output </p>\n": "<p>\u0d86\u0daf\u0dbb\u0dca\u0dc1\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the last chunk for which we haven't retrieved neighbors </p>\n": "<p>\u0d85\u0db4\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0db1\u0ddc\u0d9c\u0dad\u0dca \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Get the neighbors (with neighbor length equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca\u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 (\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf\u0d9c\u0dda \u0daf\u0dd2\u0d9c \u0dc3\u0db8\u0dcf\u0db1 <span translate=no>_^_0_^_</span>) </p>\n",
|
||||
"<p>Greedily sample the last token </p>\n": "<p>\u0d9a\u0dd1\u0daf\u0dbb\u0d9a\u0db8\u0dd2\u0db1\u0dca\u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0da7\u0ddd\u0d9a\u0db1\u0dba \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd\u0dba </p>\n",
|
||||
"<p>Hyper-parameters </p>\n": "<p>\u0d85\u0db0\u0dd2\u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca </p>\n",
|
||||
"<p>Iterate through training data </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0daf\u0dad\u0dca\u0dad \u0dc4\u0dbb\u0dc4\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load <a href=\"dataset.html\">Retro dataset</a> </p>\n": "<p><a href=\"dataset.html\">\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba</a> \u0db4\u0da7\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Load Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db4\u0da7\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dc0\u0dd9\u0dad \u0daf\u0dad\u0dca\u0dad \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move the model to the device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dc0\u0dd9\u0dad \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Move them to the same device as the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0da7\u0dc3\u0db8\u0dcf\u0db1 \u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0d9a\u0da7 \u0d92\u0dc0\u0dcf \u0d9c\u0dd9\u0db1\u0dba\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Optimize the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Print a new line </p>\n": "<p>\u0db1\u0dc0\u0dbb\u0dda\u0d9b\u0dcf\u0dc0\u0d9a\u0dca \u0db8\u0dd4\u0daf\u0dca\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Retrieve nearest neighbors </p>\n": "<p>\u0dc5\u0d9f\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Retrieve the offsets of the nearest neighbors </p>\n": "<p>\u0dc5\u0d9f\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dcf\u0dc3\u0dd3\u0db1\u0dca\u0d9c\u0dda \u0dc4\u0dd2\u0dbd\u0dc0\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Sample <span translate=no>_^_0_^_</span> tokens </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 <span translate=no>_^_0_^_</span> \u0da7\u0ddd\u0d9a\u0db1 </p>\n",
|
||||
"<p>Sample from the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc0\u0dd9\u0dad\u0dd2\u0db1\u0dca\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Sampled text </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0db4\u0dd9\u0dc5 </p>\n",
|
||||
"<p>Save models </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Save training statistics and increment the global step counter </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dbd\u0dda\u0d9b\u0db1 \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1 \u0dc3\u0dc4 \u0d9c\u0ddd\u0dbd\u0dd3\u0dba \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9a\u0dc0\u0dd4\u0db1\u0dca\u0da7\u0dbb\u0dba \u0dc0\u0dd0\u0da9\u0dd2 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>To store nearest neighbors as strings </p>\n": "<p>\u0d86\u0dc3\u0db1\u0dca\u0db1\u0dad\u0db8\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0db1\u0dd6\u0dbd\u0dca \u0dbd\u0dd9\u0dc3 \u0d9c\u0db6\u0da9\u0dcf \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>Tokenize the input </p>\n": "<p>\u0d86\u0daf\u0dcf\u0db1\u0dba\u0da7\u0ddd\u0d9a\u0dd9\u0db1\u0dca\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Tokenize the retrieved neighbors </p>\n": "<p>\u0dbd\u0db6\u0dcf\u0d9c\u0dad\u0dca \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0da7\u0ddd\u0d9a\u0dd3\u0dc3\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Train </p>\n": "<p>\u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>We need to retrieve neighbors, if there are more sampled chunks than we have already retrieved for </p>\n": "<p>\u0d85\u0db4\u0daf\u0dd0\u0db1\u0da7\u0db8\u0dad\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dd9\u0db1 \u0d87\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0da7 \u0dc0\u0da9\u0dcf \u0dc0\u0dd0\u0da9\u0dd2 \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dad\u0dd2\u0db6\u0dda \u0db1\u0db8\u0dca, \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0d9c\u0dad \u0dba\u0dd4\u0dad\u0dd4\u0dba </p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the device of the model </li>\n<li><span translate=no>_^_1_^_</span> is the <a href=\"retro.html\">Retro mode</a> </li>\n<li><span translate=no>_^_2_^_</span> is the dataloader for the <a href=\"dataset.html\">dataset with pre-retrieved neighbors</a> </li>\n<li><span translate=no>_^_3_^_</span> is the optimizer</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dda \u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba \u0dc0\u0dda </li>\n<li><span translate=no>_^_1_^_</span> <a href=\"retro.html\">\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd \u0db4\u0dca\u0dbb\u0d9a\u0dcf\u0dbb\u0dba</a> </li>\n<li><span translate=no>_^_2_^_</span> <a href=\"dataset.html\">\u0db4\u0dd9\u0dbb \u0dbd\u0db6\u0dcf \u0d9c\u0dad\u0dca \u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0db1\u0dca \u0dc3\u0db8\u0d9f \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba \u0dc0\u0dda</a> </li>\n<li><span translate=no>_^_3_^_</span> \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc0\u0dda</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the device of the model </li>\n<li><span translate=no>_^_1_^_</span> is the <a href=\"retro.html\">Retro mode</a> </li>\n<li><span translate=no>_^_2_^_</span> is the text dataset (used to get neighbor chunks) </li>\n<li><span translate=no>_^_3_^_</span> is the length of a chunk</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span> \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dda \u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba \u0dc0\u0dda </li>\n<li><span translate=no>_^_1_^_</span> <a href=\"retro.html\">\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd \u0db4\u0dca\u0dbb\u0d9a\u0dcf\u0dbb\u0dba</a> </li>\n<li><span translate=no>_^_2_^_</span> \u0dba\u0db1\u0dd4 \u0db4\u0dd9\u0dc5 \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0dd4\u0daf\u0dcf\u0dba (\u0d85\u0dc3\u0dbd\u0dca\u0dc0\u0dd0\u0dc3\u0dd2\u0dba\u0dcf \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2 \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2) </li>\n<li><span translate=no>_^_3_^_</span> \u0dba\u0db1\u0dd4 \u0d9a\u0dd4\u0da7\u0dca\u0da7\u0dd2\u0dba\u0d9a \u0daf\u0dd2\u0d9c</li></ul>\n",
|
||||
"RETRO training": "\u0dbb\u0dd9\u0da7\u0dca\u0dbb\u0ddd \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0",
|
||||
"Training RETRO model with Tiny Shakespeare dataset": "\u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0dc3\u0db8\u0d9f \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 RETRO \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba"
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"<h1>RETRO training</h1>\n<p>This is the training code for <a href=\"index.html\">RETRO</a>.</p>\n": "<h1>\u590d\u53e4\u8bad\u7ec3</h1>\n<p>\u8fd9\u662f RETRO \u7684\u8bad\u7ec3<a href=\"index.html\">\u4ee3\u7801</a>\u3002</p>\n",
|
||||
"<h2>Create and train a small model</h2>\n": "<h2>\u521b\u5efa\u548c\u8bad\u7ec3\u4e00\u4e2a\u5c0f\u6a21\u578b</h2>\n",
|
||||
"<h2>Retro trainer</h2>\n": "<h2>\u590d\u53e4\u6559\u7ec3</h2>\n",
|
||||
"<h2>Sampler</h2>\n<p>This class greedily samples from a model.</p>\n": "<h2>\u91c7\u6837\u5668</h2>\n<p>\u8fd9\u4e2a\u7c7b\u8d2a\u5a6a\u5730\u4ece\u6a21\u578b\u4e2d\u62bd\u6837\u3002</p>\n",
|
||||
"<h3>Retrieve nearest neighbors of a given chunk</h3>\n": "<h3>\u68c0\u7d22\u7ed9\u5b9a\u533a\u5757\u7684\u6700\u8fd1\u90bb\u57df</h3>\n",
|
||||
"<h3>Sample text from the given prompt</h3>\n": "<h3>\u6765\u81ea\u7ed9\u5b9a\u63d0\u793a\u7684\u793a\u4f8b\u6587\u672c</h3>\n",
|
||||
"<h3>Train the model for an epoch</h3>\n": "<h3>\u4e3a\u4e00\u4e2a\u65f6\u4ee3\u8bad\u7ec3\u6a21\u578b</h3>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><a href=\"database.html\">Retro index</a> </p>\n": "<p><a href=\"database.html\">\u590d\u53e4\u6307\u6570</a></p>\n",
|
||||
"<p>Add the sampled token text to the prompt and sample text </p>\n": "<p>\u5c06\u91c7\u6837\u6807\u8bb0\u6587\u672c\u6dfb\u52a0\u5230\u63d0\u793a\u548c\u793a\u4f8b\u6587\u672c\u4e2d</p>\n",
|
||||
"<p>Backward pass </p>\n": "<p>\u5411\u540e\u4f20\u7403</p>\n",
|
||||
"<p>Calculate loss </p>\n": "<p>\u8ba1\u7b97\u635f\u5931</p>\n",
|
||||
"<p>Clear the gradients </p>\n": "<p>\u6e05\u9664\u6e10\u53d8</p>\n",
|
||||
"<p>Create an experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Create dataloader </p>\n": "<p>\u521b\u5efa\u6570\u636e\u52a0\u8f7d\u5668</p>\n",
|
||||
"<p>Create the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u521b\u5efa<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Create the model </p>\n": "<p>\u521b\u5efa\u6a21\u578b</p>\n",
|
||||
"<p>Create the nearest neighbor encoder </p>\n": "<p>\u521b\u5efa\u6700\u8fd1\u90bb\u7f16\u7801\u5668</p>\n",
|
||||
"<p>Create the optimizer </p>\n": "<p>\u521b\u5efa\u4f18\u5316\u5668</p>\n",
|
||||
"<p>Forward pass </p>\n": "<p>\u5411\u524d\u4f20\u7403</p>\n",
|
||||
"<p>GPU device </p>\n": "<p>GPU \u8bbe\u5907</p>\n",
|
||||
"<p>Get model output </p>\n": "<p>\u83b7\u53d6\u6a21\u578b\u8f93\u51fa</p>\n",
|
||||
"<p>Get the last chunk for which we haven't retrieved neighbors </p>\n": "<p>\u83b7\u53d6\u6211\u4eec\u5c1a\u672a\u68c0\u7d22\u5230\u90bb\u5c45\u7684\u6700\u540e\u4e00\u4e2a\u533a\u5757</p>\n",
|
||||
"<p>Get the neighbors (with neighbor length equal to <span translate=no>_^_0_^_</span>) </p>\n": "<p>\u83b7\u53d6\u90bb\u5c45\uff08\u90bb\u5c45\u957f\u5ea6\u7b49\u4e8e<span translate=no>_^_0_^_</span>\uff09</p>\n",
|
||||
"<p>Greedily sample the last token </p>\n": "<p>\u8d2a\u5a6a\u5730\u62bd\u53d6\u6700\u540e\u4e00\u4e2a\u4ee3\u5e01</p>\n",
|
||||
"<p>Hyper-parameters </p>\n": "<p>\u8d85\u53c2\u6570</p>\n",
|
||||
"<p>Iterate through training data </p>\n": "<p>\u904d\u5386\u8bad\u7ec3\u6570\u636e</p>\n",
|
||||
"<p>Load <a href=\"dataset.html\">Retro dataset</a> </p>\n": "<p>\u52a0\u8f7d<a href=\"dataset.html\">\u590d\u53e4\u6570\u636e\u96c6</a></p>\n",
|
||||
"<p>Load Tiny Shakespeare dataset </p>\n": "<p>\u52a0\u8f7d\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
|
||||
"<p>Move data to the device </p>\n": "<p>\u5c06\u6570\u636e\u79fb\u52a8\u5230\u8bbe\u5907</p>\n",
|
||||
"<p>Move the model to the device </p>\n": "<p>\u5c06\u6a21\u578b\u79fb\u5230\u8bbe\u5907\u4e0a</p>\n",
|
||||
"<p>Move them to the same device as the model </p>\n": "<p>\u5c06\u5b83\u4eec\u79fb\u5230\u4e0e\u6a21\u578b\u76f8\u540c\u7684\u8bbe\u5907\u4e0a</p>\n",
|
||||
"<p>Optimize the model </p>\n": "<p>\u4f18\u5316\u6a21\u578b</p>\n",
|
||||
"<p>Print a new line </p>\n": "<p>\u6253\u5370\u4e00\u6761\u65b0\u884c</p>\n",
|
||||
"<p>Retrieve nearest neighbors </p>\n": "<p>\u68c0\u7d22\u6700\u8fd1\u7684\u90bb\u5c45</p>\n",
|
||||
"<p>Retrieve the offsets of the nearest neighbors </p>\n": "<p>\u68c0\u7d22\u6700\u8fd1\u90bb\u70b9\u7684\u504f\u79fb\u91cf</p>\n",
|
||||
"<p>Sample <span translate=no>_^_0_^_</span> tokens </p>\n": "<p><span translate=no>_^_0_^_</span>\u4ee3\u5e01\u6837\u672c</p>\n",
|
||||
"<p>Sample from the <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6837\u672c\u6765\u81ea<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Sampled text </p>\n": "<p>\u6837\u672c\u6587\u672c</p>\n",
|
||||
"<p>Save models </p>\n": "<p>\u4fdd\u5b58\u6a21\u578b</p>\n",
|
||||
"<p>Save training statistics and increment the global step counter </p>\n": "<p>\u4fdd\u5b58\u8bad\u7ec3\u7edf\u8ba1\u6570\u636e\u5e76\u589e\u52a0\u5168\u5c40\u6b65\u6570\u8ba1\u6570\u5668</p>\n",
|
||||
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>To store nearest neighbors as strings </p>\n": "<p>\u5c06\u6700\u8fd1\u90bb\u5b58\u50a8\u4e3a\u5b57\u7b26\u4e32</p>\n",
|
||||
"<p>Tokenize the input </p>\n": "<p>\u5bf9\u8f93\u5165\u8fdb\u884c\u6807\u8bb0\u5316</p>\n",
|
||||
"<p>Tokenize the retrieved neighbors </p>\n": "<p>\u6807\u8bb0\u68c0\u7d22\u5230\u7684\u90bb\u5c45</p>\n",
|
||||
"<p>Train </p>\n": "<p>\u706b\u8f66</p>\n",
|
||||
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
|
||||
"<p>We need to retrieve neighbors, if there are more sampled chunks than we have already retrieved for </p>\n": "<p>\u5982\u679c\u91c7\u6837\u5757\u6bd4\u6211\u4eec\u5df2\u7ecf\u68c0\u7d22\u5230\u7684\u8981\u591a\uff0c\u6211\u4eec\u5c31\u9700\u8981\u68c0\u7d22\u90bb\u5c45</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the device of the model </li>\n<li><span translate=no>_^_1_^_</span> is the <a href=\"retro.html\">Retro mode</a> </li>\n<li><span translate=no>_^_2_^_</span> is the dataloader for the <a href=\"dataset.html\">dataset with pre-retrieved neighbors</a> </li>\n<li><span translate=no>_^_3_^_</span> is the optimizer</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8be5\u578b\u53f7\u7684\u8bbe\u5907</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u590d<a href=\"retro.html\">\u53e4\u6a21\u5f0f</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f<a href=\"dataset.html\">\u5177\u6709\u9884\u68c0\u7d22\u90bb\u57df\u7684\u6570\u636e\u96c6\u7684\u6570\u636e</a>\u52a0\u8f7d\u5668</li>\n<li><span translate=no>_^_3_^_</span>\u662f\u4f18\u5316\u5668</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> is the device of the model </li>\n<li><span translate=no>_^_1_^_</span> is the <a href=\"retro.html\">Retro mode</a> </li>\n<li><span translate=no>_^_2_^_</span> is the text dataset (used to get neighbor chunks) </li>\n<li><span translate=no>_^_3_^_</span> is the length of a chunk</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u662f\u8be5\u578b\u53f7\u7684\u8bbe\u5907</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u590d<a href=\"retro.html\">\u53e4\u6a21\u5f0f</a></li>\n<li><span translate=no>_^_2_^_</span>\u662f\u6587\u672c\u6570\u636e\u96c6\uff08\u7528\u4e8e\u83b7\u53d6\u76f8\u90bb\u6570\u636e\u5757\uff09</li>\n<li><span translate=no>_^_3_^_</span>\u662f\u533a\u5757\u7684\u957f\u5ea6</li></ul>\n",
|
||||
"RETRO training": "\u590d\u53e4\u8bad\u7ec3",
|
||||
"Training RETRO model with Tiny Shakespeare dataset": "\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6\u8bad\u7ec3 RETRO \u6a21\u578b"
|
||||
}
|
||||
Reference in New Issue
Block a user