wehub/labmlai--annotated_deep_learning_paper_implementations
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:19:01 +08:00
commit 3b90d1192f
2172 changed files with 594509 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
translate_cache/transformers/alibi/__init__.ja.json
+35
View File
@@ -0,0 +1,35 @@
{
"<h1>Attention with Linear Biases (ALiBi)</h1>\n<p>This is an implementation of Attention with Linear Biases (ALiBi) from the paper <a href=\"https://arxiv.org/abs/2108.12409\">Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation</a>.</p>\n<p>This replaces positional encodings with biases added to attention scores (attention logits, before the softmax). This is a relative scheme tested on autoregressive tasks, and the bias is higher for closeby tokens and lower for far-away tokens. The biases decrease linearly in the log scale (because it&#x27;s before the softmax) and each head has a different slope.</p>\n<p>Here&#x27;s the attention formula for <span translate=no>_^_0_^_</span>-th token,</p>\n<span translate=no>_^_1_^_</span><p>where <span translate=no>_^_2_^_</span> is the query of the <span translate=no>_^_3_^_</span>-th token, <span translate=no>_^_4_^_</span> are the keys up to <span translate=no>_^_5_^_</span>, and <span translate=no>_^_6_^_</span> the number of features per head. Note that the above equality halts because <span translate=no>_^_7_^_</span> is invariant to translations (you can add any constant to all elements without changing the result).</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a ALiBi model.</p>\n": "<h1>\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f (AliBi)</h1>\n<p>\u3053\u308c\u306f\u3001\u300c<a href=\"https://arxiv.org/abs/2108.12409\">\u30c8\u30ec\u30a4\u30f3\u30b7\u30e7\u30fc\u30c8\u3001\u30c6\u30b9\u30c8\u30ed\u30f3\u30b0\u300d\u3068\u3044\u3046\u8ad6\u6587\u306e\u300c\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f\uff08AliBi\uff09\u300d\u306e\u5b9f\u88c5\u3067\u3059\u3002\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f\u306b\u3088\u308a\u3001\u5165\u529b\u306e\u9577\u3055\u306e\u63a8\u5b9a\u304c\u53ef\u80fd\u306b\u306a\u308a\u307e\u3059</a>\u3002</p>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\u304c\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2\uff08\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9\u306e\u524d\u306e\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ed\u30b8\u30c3\u30c8\uff09\u306b\u30d0\u30a4\u30a2\u30b9\u304c\u52a0\u308f\u3063\u305f\u3082\u306e\u306b\u7f6e\u304d\u63db\u308f\u308a\u307e\u3059\u3002\u3053\u308c\u306f\u81ea\u5df1\u56de\u5e30\u30bf\u30b9\u30af\u3067\u30c6\u30b9\u30c8\u3055\u308c\u305f\u76f8\u5bfe\u7684\u306a\u30b9\u30ad\u30fc\u30e0\u3067\u3001\u8fd1\u304f\u306b\u3042\u308b\u30c8\u30fc\u30af\u30f3\u306e\u65b9\u304c\u30d0\u30a4\u30a2\u30b9\u304c\u5927\u304d\u304f\u3001\u9060\u3044\u30c8\u30fc\u30af\u30f3\u306e\u65b9\u304c\u30d0\u30a4\u30a2\u30b9\u304c\u4f4e\u304f\u306a\u308a\u307e\u3059\u3002\u5bfe\u6570\u30b9\u30b1\u30fc\u30eb\u3067\u306f\uff08\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9\u306e\u524d\u306a\u306e\u3067\uff09\u30d0\u30a4\u30a2\u30b9\u306f\u76f4\u7dda\u7684\u306b\u6e1b\u5c11\u3057\u3001\u5404\u30d8\u30c3\u30c9\u306e\u50be\u304d\u306f\u7570\u306a\u308a\u307e\u3059</p>\u3002\n<p><span translate=no>_^_0_^_</span>-th \u30c8\u30fc\u30af\u30f3\u306e\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30d5\u30a9\u30fc\u30df\u30e5\u30e9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059\u3002</p>\n<span translate=no>_^_1_^_</span><p>\u3053\u3053\u3067\u3001<span translate=no>_^_2_^_</span>\u306f <span translate=no>_^_3_^_</span>-th \u30c8\u30fc\u30af\u30f3\u306e\u30af\u30a8\u30ea\u3001<span translate=no>_^_4_^_</span>\u307e\u3067\u306e\u30ad\u30fc<span translate=no>_^_5_^_</span>\u3001<span translate=no>_^_6_^_</span>\u304a\u3088\u3073\u30d8\u30c3\u30c9\u3042\u305f\u308a\u306e\u30d5\u30a3\u30fc\u30c1\u30e3\u6570\u3067\u3059\u3002<span translate=no>_^_7_^_</span>\u4e0a\u8a18\u306e\u7b49\u5f0f\u306f\u5909\u63db\u306b\u4e0d\u5909\u3067\u3042\u308b\u305f\u3081\u4e2d\u6b62\u3055\u308c\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044 (\u7d50\u679c\u3092\u5909\u66f4\u305b\u305a\u306b\u3059\u3079\u3066\u306e\u8981\u7d20\u306b\u4efb\u610f\u306e\u5b9a\u6570\u3092\u8ffd\u52a0\u3067\u304d\u307e\u3059</p>)\u3002\n<p>AliBi <a href=\"experiment.html\">\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b3\u30fc\u30c9\u306f\u6b21\u306e\u3068\u304a\u308a\u3067\u3059</a>\u3002</p>\n",
"<h2>Attention with Linear Biases (ALiBi)</h2>\n<p>We override <a href=\"../mha.html\">Multi-Head Attention</a>.</p>\n": "<h2>\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f (AliBi)</h2>\n<p><a href=\"../mha.html\">\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u3092\u7121\u52b9\u306b\u3057\u307e\u3059</a>\u3002</p>\n",
"<h2>Calculate the attention biases matrix</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the number of heads in the attention layer </li>\n<li><span translate=no>_^_1_^_</span> is the attention mask of shape <span translate=no>_^_2_^_</span></li></ul>\n<p>This returns a matrix of shape <span translate=no>_^_3_^_</span> with ALiBi attention biases.</p>\n": "<h2>\u6ce8\u610f\u30d0\u30a4\u30a2\u30b9\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u306e\u8a08\u7b97</h2>\n<ul><li><span translate=no>_^_0_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u306e\u30d8\u30c3\u30c9\u6570\u3067\u3059</li>\n<li><span translate=no>_^_1_^_</span>\u30b7\u30a7\u30a4\u30d7\u306e\u6ce8\u610f\u30de\u30b9\u30af\u3067\u3059 <span translate=no>_^_2_^_</span></li></ul>\n<p>\u3053\u308c\u306b\u3088\u308a\u3001AliBi <span translate=no>_^_3_^_</span> \u306e\u6ce8\u610f\u30d0\u30a4\u30a2\u30b9\u304c\u5165\u3063\u305f\u5f62\u72b6\u306e\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u304c\u8fd4\u3055\u308c\u307e\u3059\u3002</p>\n",
"<h2>Get head-specific slope <span translate=no>_^_0_^_</span> for each head</h2>\n<ul><li><span translate=no>_^_1_^_</span> is the number of heads in the attention layer <span translate=no>_^_2_^_</span></li></ul>\n<p>The slope for first head is</p>\n<p><span translate=no>_^_3_^_</span></p>\n<p>The slopes for the rest of the heads are in a geometric series with a ratio same as above.</p>\n<p>For instance when the number of heads is <span translate=no>_^_4_^_</span> the slopes are <span translate=no>_^_5_^_</span></p>\n": "<h2><span translate=no>_^_0_^_</span>\u5404\u982d\u90e8\u306e\u982d\u90e8\u56fa\u6709\u306e\u52fe\u914d\u3092\u53d6\u5f97</h2>\n<ul><li><span translate=no>_^_1_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30ec\u30a4\u30e4\u30fc\u306e\u30d8\u30c3\u30c9\u6570\u3067\u3059 <span translate=no>_^_2_^_</span></li></ul>\n<p>1 \u756a\u76ee\u306e\u30d8\u30c3\u30c9\u306e\u52fe\u914d\u306f</p>\n<p><span translate=no>_^_3_^_</span></p>\n<p>\u6b8b\u308a\u306e\u30d8\u30c3\u30c9\u306e\u52fe\u914d\u306f\u5e7e\u4f55\u5b66\u7684\u306b\u9023\u7d9a\u3057\u3066\u304a\u308a\u3001\u305d\u306e\u6bd4\u7387\u306f\u4e0a\u8a18\u3068\u540c\u3058\u3067\u3059\u3002</p>\n<p>\u305f\u3068\u3048\u3070\u3001\u30d8\u30c3\u30c9\u306e\u6570\u304c\u306e\u5834\u5408\u3001<span translate=no>_^_4_^_</span>\u30b9\u30ed\u30fc\u30d7\u306f <span translate=no>_^_5_^_</span></p>\n",
"<p> </p>\n": "<p></p>\n",
"<p> <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> are the tensors that store collection of <em>query</em>, <em>key</em> and <em>value</em> vectors. They have shape <span translate=no>_^_3_^_</span>.</p>\n<p><span translate=no>_^_4_^_</span> has shape <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> indicates whether for batch <span translate=no>_^_7_^_</span>, query at position <span translate=no>_^_8_^_</span> has access to key-value at position <span translate=no>_^_9_^_</span>.</p>\n": "<p><span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u304a\u3088\u3073\u306f\u3001<em>\u30af\u30a8\u30ea</em>\u3001<em>\u30ad\u30fc</em>\u3001<em>\u304a\u3088\u3073\u5024\u306e\u30d9\u30af\u30c8\u30eb\u306e\u30b3\u30ec\u30af\u30b7\u30e7\u30f3\u3092\u683c\u7d0d\u3059\u308b\u30c6\u30f3\u30bd\u30eb\u3067\u3059</em>\u3002\u5f62\u304c\u3042\u308a\u307e\u3059<span translate=no>_^_3_^_</span>\u3002</p>\n<p><span translate=no>_^_4_^_</span><span translate=no>_^_5_^_</span>\u5f62\u72b6\u304c\u3042\u308a\u3001\u30d0\u30c3\u30c1\u306e\u5834\u5408<span translate=no>_^_7_^_</span>\u3001<span translate=no>_^_6_^_</span><span translate=no>_^_8_^_</span>\u305d\u306e\u4f4d\u7f6e\u306e\u30af\u30a8\u30ea\u304c\u305d\u306e\u4f4d\u7f6e\u306e\u30ad\u30fc\u5024\u306b\u30a2\u30af\u30bb\u30b9\u3067\u304d\u308b\u304b\u3069\u3046\u304b\u3092\u793a\u3057\u307e\u3059\u3002<span translate=no>_^_9_^_</span></p>\n",
"<p> Simple test function to see the slopes.</p>\n": "<p>\u30b9\u30ed\u30fc\u30d7\u3092\u78ba\u8a8d\u3067\u304d\u308b\u7c21\u5358\u306a\u30c6\u30b9\u30c8\u6a5f\u80fd\u3002</p>\n",
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
"<p><span translate=no>_^_0_^_</span> Note that we take steps by <span translate=no>_^_1_^_</span> to avoid slopes added previously. </p>\n": "<p><span translate=no>_^_0_^_</span>\u306a\u304a\u3001<span translate=no>_^_1_^_</span>\u4ee5\u524d\u306b\u30b9\u30ed\u30fc\u30d7\u304c\u8ffd\u52a0\u3055\u308c\u306a\u3044\u3088\u3046\u306b\u5bfe\u7b56\u3092\u8b1b\u3058\u3066\u3044\u307e\u3059\u3002</p>\n",
"<p><span translate=no>_^_0_^_</span> attention along the key sequence dimension <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u30ad\u30fc\u30b7\u30fc\u30b1\u30f3\u30b9\u6b21\u5143\u306b\u6cbf\u3063\u3066\u6ce8\u76ee <span translate=no>_^_1_^_</span></p>\n",
"<p><span translate=no>_^_0_^_</span> has shape <a href=\"seq_len, seq_len, 1, 1\">seq_len, seq_len, 1, 1</a> </p>\n": "<p><span translate=no>_^_0_^_</span><a href=\"seq_len, seq_len, 1, 1\">\u56f3\u5f62\u306f\u9023\u756a\u3001\u9023\u756a\u30011\u3001</a> 1\u3067\u3059</p>\n",
"<p><span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> have shape <span translate=no>_^_3_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u305d\u3057\u3066\u5f62\u304c\u3042\u308b <span translate=no>_^_3_^_</span></p>\n",
"<p>ALiBi only works with causal masks. </p>\n": "<p>AliBi \u306f\u56e0\u679c\u30de\u30b9\u30af\u3067\u306e\u307f\u6a5f\u80fd\u3057\u307e\u3059\u3002</p>\n",
"<p>Add AliBi biases to attention scores. ALiBi biases has shape <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> has shape <span translate=no>_^_2_^_</span> </p>\n": "<p>AliBi \u30d0\u30a4\u30a2\u30b9\u3092\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2\u306b\u8ffd\u52a0\u3057\u307e\u3059\u3002AliBi <span translate=no>_^_0_^_</span> <span translate=no>_^_1_^_</span> \u30d0\u30a4\u30a2\u30b9\u306b\u306f\u5f62\u3068\u5f62\u304c\u3042\u308b <span translate=no>_^_2_^_</span></p>\n",
"<p>Add head dimension to mask and check its shape. </p>\n": "<p>\u30de\u30b9\u30af\u306b\u982d\u90e8\u306e\u5bf8\u6cd5\u3092\u8ffd\u52a0\u3057\u3001\u5f62\u72b6\u3092\u78ba\u8a8d\u3057\u307e\u3059\u3002</p>\n",
"<p>Apply dropout </p>\n": "<p>\u30c9\u30ed\u30c3\u30d7\u30a2\u30a6\u30c8\u3092\u9069\u7528</p>\n",
"<p>Apply mask </p>\n": "<p>\u30de\u30b9\u30af\u3092\u9069\u7528</p>\n",
"<p>Calculate distances <span translate=no>_^_0_^_</span> Here we calculate the distances using the mask.</p>\n<p>Since it&#x27;s causal mask we can just use <span translate=no>_^_1_^_</span> too. <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u8ddd\u96e2\u306e\u8a08\u7b97\u3053\u3053\u3067\u306f\u30de\u30b9\u30af\u3092\u4f7f\u3063\u3066\u8ddd\u96e2\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002</p>\n<p><span translate=no>_^_1_^_</span>\u30ab\u30b8\u30e5\u30a2\u30eb\u30de\u30b9\u30af\u306a\u306e\u3067\u305d\u306e\u307e\u307e\u4f7f\u3048\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n",
"<p>Compute attention scores <span translate=no>_^_0_^_</span>. This gives a tensor of shape <span translate=no>_^_1_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span>\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30b9\u30b3\u30a2\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002<span translate=no>_^_1_^_</span>\u3053\u308c\u306b\u3088\u308a\u5f62\u72b6\u306e\u30c6\u30f3\u30bd\u30eb\u304c\u5f97\u3089\u308c\u307e\u3059</p>\u3002\n",
"<p>Concatenate multiple heads </p>\n": "<p>\u8907\u6570\u306e\u30d8\u30c3\u30c9\u3092\u9023\u7d50</p>\n",
"<p>Concatenate the slopes with the remaining slopes. </p>\n": "<p>\u30b9\u30ed\u30fc\u30d7\u3092\u6b8b\u308a\u306e\u30b9\u30ed\u30fc\u30d7\u3068\u9023\u7d50\u3057\u307e\u3059\u3002</p>\n",
"<p>Create AliBi biases if it&#x27;s not cached </p>\n": "<p>\u30ad\u30e3\u30c3\u30b7\u30e5\u3055\u308c\u3066\u3044\u306a\u3044\u5834\u5408\u306fAliBi\u30d0\u30a4\u30a2\u30b9\u3092\u4f5c\u6210\u3059\u308b</p>\n",
"<p>Get slopes <span translate=no>_^_0_^_</span> for each head </p>\n": "<p><span translate=no>_^_0_^_</span>\u5404\u30d8\u30c3\u30c9\u306e\u30b9\u30ed\u30fc\u30d7\u3092\u53d6\u5f97</p>\n",
"<p>Get the closest power of 2 to <span translate=no>_^_0_^_</span>. If <span translate=no>_^_1_^_</span> is not a power of 2, then we first calculate slopes to the closest (smaller) power of 2, and then add the remaining slopes. </p>\n": "<p>2 <span translate=no>_^_0_^_</span> \u306e\u7d2f\u4e57\u306b\u6700\u3082\u8fd1\u3044\u3082\u306e\u3092\u6c42\u3081\u307e\u3059\u3002\u304c 2 <span translate=no>_^_1_^_</span> \u306e\u7d2f\u4e57\u3067\u306a\u3044\u5834\u5408\u306f\u3001\u307e\u305a 2 \u306b\u6700\u3082\u8fd1\u3044 (\u5c0f\u3055\u306a) \u7d2f\u4e57\u307e\u3067\u306e\u52fe\u914d\u3092\u8a08\u7b97\u3057\u3001\u6b21\u306b\u6b8b\u308a\u306e\u52fe\u914d\u3092\u52a0\u7b97\u3057\u307e\u3059</p>\u3002\n",
"<p>If <span translate=no>_^_0_^_</span> is not a power of 2, then we add the remaining slopes. We calculate the remaining slopes for <span translate=no>_^_1_^_</span> (avoiding slopes added previously). And pick the slopes upto <span translate=no>_^_2_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span>\u304c 2 \u306e\u7d2f\u4e57\u3067\u306a\u3044\u5834\u5408\u306f\u3001\u6b8b\u308a\u306e\u52fe\u914d\u3092\u52a0\u7b97\u3057\u307e\u3059\u3002\u6b8b\u308a\u306e\u52fe\u914d\u3092\u8a08\u7b97\u3057\u307e\u3059 <span translate=no>_^_1_^_</span> (\u4ee5\u524d\u306b\u8ffd\u52a0\u3055\u308c\u305f\u52fe\u914d\u306f\u9664\u304d\u307e\u3059)\u3002\u305d\u3057\u3066\u3001<span translate=no>_^_2_^_</span>\u4e0a\u306e\u659c\u9762\u3092\u9078\u3093\u3067\u304f\u3060\u3055\u3044</p>.\n",
"<p>Multiply by values <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5024\u306b\u3088\u308b\u4e57\u7b97 <span translate=no>_^_0_^_</span></p>\n",
"<p>Multiply them pair-wise to get the AliBi bias matrix </p>\n": "<p>\u305d\u308c\u3089\u3092\u30da\u30a2\u3054\u3068\u306b\u4e57\u7b97\u3057\u3066\u3001AliBi \u30d0\u30a4\u30a2\u30b9\u30de\u30c8\u30ea\u30c3\u30af\u30b9\u3092\u6c42\u3081\u307e\u3059\u3002</p>\n",
"<p>Output layer </p>\n": "<p>\u51fa\u529b\u30ec\u30a4\u30e4\u30fc</p>\n",
"<p>Prepare <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> for attention computation. These will then have shape <span translate=no>_^_3_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u6ce8\u610f\u529b\u8a08\u7b97\u306e\u6e96\u5099\u3092\u3057\u3066<span translate=no>_^_3_^_</span>\u3053\u308c\u3067\u5f62\u304c\u3067\u304d\u3042\u304c\u308a\u307e\u3059\u3002</p>\n",
"<p>Scale scores <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30b1\u30fc\u30eb\u30b9\u30b3\u30a2 <span translate=no>_^_0_^_</span></p>\n",
"<p>To cache AliBi the biases </p>\n": "<p>AliBi \u306b\u30d0\u30a4\u30a2\u30b9\u3092\u30ad\u30e3\u30c3\u30b7\u30e5\u3059\u308b\u306b\u306f</p>\n",
"Attention with Linear Biases (ALiBi)": "\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f (AliBi)",
"Documented implementation with explanations of Attention with Linear Biases (ALiBi)": "\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\u306b\u3088\u308b\u6ce8\u610f\uff08AliBi\uff09\u306e\u8aac\u660e\u3092\u542b\u3080\u6587\u66f8\u5316\u3055\u308c\u305f\u5b9f\u88c5"
}
translate_cache/transformers/alibi/__init__.si.json
+35
View File
File diff suppressed because one or more lines are too long
translate_cache/transformers/alibi/__init__.zh.json
+35
View File
@@ -0,0 +1,35 @@
{
"<h1>Attention with Linear Biases (ALiBi)</h1>\n<p>This is an implementation of Attention with Linear Biases (ALiBi) from the paper <a href=\"https://arxiv.org/abs/2108.12409\">Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation</a>.</p>\n<p>This replaces positional encodings with biases added to attention scores (attention logits, before the softmax). This is a relative scheme tested on autoregressive tasks, and the bias is higher for closeby tokens and lower for far-away tokens. The biases decrease linearly in the log scale (because it&#x27;s before the softmax) and each head has a different slope.</p>\n<p>Here&#x27;s the attention formula for <span translate=no>_^_0_^_</span>-th token,</p>\n<span translate=no>_^_1_^_</span><p>where <span translate=no>_^_2_^_</span> is the query of the <span translate=no>_^_3_^_</span>-th token, <span translate=no>_^_4_^_</span> are the keys up to <span translate=no>_^_5_^_</span>, and <span translate=no>_^_6_^_</span> the number of features per head. Note that the above equality halts because <span translate=no>_^_7_^_</span> is invariant to translations (you can add any constant to all elements without changing the result).</p>\n<p>Here is <a href=\"experiment.html\">the training code</a> for a ALiBi model.</p>\n": "<h1>\u6ce8\u610f\u7ebf\u6027\u504f\u5dee (AliBI)</h1>\n<p>\u8fd9\u662f\u300aT <a href=\"https://arxiv.org/abs/2108.12409\">rain Short\uff0cTest Long\uff1a\u4f7f\u7528\u7ebf\u6027\u504f\u5dee\u7684\u6ce8\u610f\u529b\u5b9e\u73b0\u8f93\u5165\u957f\u5ea6\u5916\u63a8\u300b\u4e00\u6587\u4e2d\u7684 \u201c\u4f7f\u7528\u7ebf\u6027\u504f\u5dee\u6ce8\u610f\u529b</a> (AliBI)\u201d \u7684\u5b9e\u73b0\u3002</p>\n<p>\u8fd9\u5c06\u7528\u5728\u6ce8\u610f\u529b\u5206\u6570\u4e2d\u6dfb\u52a0\u504f\u5dee\uff08\u6ce8\u610f\u529b\u5bf9\u6570\uff0c\u5728 softmax \u4e4b\u524d\uff09\u53d6\u4ee3\u4f4d\u7f6e\u7f16\u7801\u3002\u8fd9\u662f\u4e00\u79cd\u5728\u81ea\u56de\u5f52\u4efb\u52a1\u4e0a\u6d4b\u8bd5\u7684\u76f8\u5bf9\u65b9\u6848\uff0ccloseby\u4ee3\u5e01\u7684\u504f\u5dee\u66f4\u9ad8\uff0c\u800c\u9065\u8fdc\u7684\u4ee3\u5e01\u7684\u504f\u5dee\u66f4\u4f4e\u3002\u504f\u5dee\u5728\u5bf9\u6570\u6807\u5ea6\u4e2d\u5448\u7ebf\u6027\u51cf\u5c0f\uff08\u56e0\u4e3a\u5b83\u5728softmax\u4e4b\u524d\uff09\uff0c\u5e76\u4e14\u6bcf\u4e2a\u5934\u90e8\u90fd\u6709\u4e0d\u540c\u7684\u659c\u7387\u3002</p>\n<p>\u8fd9\u662f<span translate=no>_^_0_^_</span>\u7b2c-th \u4ee3\u5e01\u7684\u6ce8\u610f\u529b\u516c\u5f0f\uff0c</p>\n<span translate=no>_^_1_^_</span><p>\u5176\u4e2d\uff0c<span translate=no>_^_2_^_</span>\u662f<span translate=no>_^_3_^_</span>\u7b2c-th \u4e2a\u4ee4\u724c\u7684\u67e5\u8be2\uff0c\u6700\u5927<span translate=no>_^_4_^_</span>\u662f\u5bc6\u94a5\u6570\u4ee5\u53ca<span translate=no>_^_6_^_</span>\u6bcf\u4e2a\u6807\u5934\u7684\u8981<span translate=no>_^_5_^_</span>\u7d20\u6570\u3002\u8bf7\u6ce8\u610f\uff0c\u4e0a\u8ff0\u7b49\u5f0f\u4e4b\u6240\u4ee5\u505c\u6b62\uff0c\u662f\u56e0\u4e3a\u7ffb\u8bd1\u662f\u4e0d\u53d8<span translate=no>_^_7_^_</span>\u7684\uff08\u60a8\u53ef\u4ee5\u5728\u4e0d\u66f4\u6539\u7ed3\u679c\u7684\u60c5\u51b5\u4e0b\u5411\u6240\u6709\u5143\u7d20\u6dfb\u52a0\u4efb\u4f55\u5e38\u91cf\uff09\u3002</p>\n<p><a href=\"experiment.html\">\u4ee5\u4e0b\u662f AliBi \u6a21\u578b\u7684\u8bad\u7ec3\u4ee3\u7801</a>\u3002</p>\n",
"<h2>Attention with Linear Biases (ALiBi)</h2>\n<p>We override <a href=\"../mha.html\">Multi-Head Attention</a>.</p>\n": "<h2>\u6ce8\u610f\u7ebf\u6027\u504f\u5dee (AliBI)</h2>\n<p>\u6211\u4eec\u8986\u76d6<a href=\"../mha.html\">\u591a\u5934\u6ce8\u610f\u529b</a>\u3002</p>\n",
"<h2>Calculate the attention biases matrix</h2>\n<ul><li><span translate=no>_^_0_^_</span> is the number of heads in the attention layer </li>\n<li><span translate=no>_^_1_^_</span> is the attention mask of shape <span translate=no>_^_2_^_</span></li></ul>\n<p>This returns a matrix of shape <span translate=no>_^_3_^_</span> with ALiBi attention biases.</p>\n": "<h2>\u8ba1\u7b97\u6ce8\u610f\u529b\u504f\u5dee\u77e9\u9635</h2>\n<ul><li><span translate=no>_^_0_^_</span>\u662f\u6ce8\u610f\u5c42\u4e2d\u7684\u5934\u90e8\u6570\u91cf</li>\n<li><span translate=no>_^_1_^_</span>\u662f\u5f62\u72b6\u7684\u6ce8\u610f\u529b\u9762\u5177<span translate=no>_^_2_^_</span></li></ul>\n<p>\u8fd9\u5c06\u8fd4\u56de\u4e00\u4e2a<span translate=no>_^_3_^_</span>\u5177\u6709 AliBi \u6ce8\u610f\u529b\u504f\u5dee\u7684\u5f62\u72b6\u77e9\u9635\u3002</p>\n",
"<h2>Get head-specific slope <span translate=no>_^_0_^_</span> for each head</h2>\n<ul><li><span translate=no>_^_1_^_</span> is the number of heads in the attention layer <span translate=no>_^_2_^_</span></li></ul>\n<p>The slope for first head is</p>\n<p><span translate=no>_^_3_^_</span></p>\n<p>The slopes for the rest of the heads are in a geometric series with a ratio same as above.</p>\n<p>For instance when the number of heads is <span translate=no>_^_4_^_</span> the slopes are <span translate=no>_^_5_^_</span></p>\n": "<h2><span translate=no>_^_0_^_</span>\u4e3a\u6bcf\u4e2a\u5934\u90e8\u83b7\u53d6\u7279\u5b9a\u4e8e\u5934\u90e8\u7684\u659c\u7387</h2>\n<ul><li><span translate=no>_^_1_^_</span>\u662f\u6ce8\u610f\u5c42\u4e2d\u7684\u5934\u90e8\u6570\u91cf<span translate=no>_^_2_^_</span></li></ul>\n<p>\u7b2c\u4e00\u4e2a\u5934\u7684\u659c\u7387\u662f</p>\n<p><span translate=no>_^_3_^_</span></p>\n<p>\u5176\u4f59\u5934\u90e8\u7684\u659c\u7387\u4e3a\u51e0\u4f55\u5e8f\u5217\uff0c\u5176\u6bd4\u4f8b\u4e0e\u4e0a\u9762\u76f8\u540c\u3002</p>\n<p>\u4f8b\u5982\uff0c\u5f53\u5934\u6570\u4e3a\u65f6<span translate=no>_^_4_^_</span>\uff0c\u659c\u7387\u4e3a<span translate=no>_^_5_^_</span></p>\n",
"<p> </p>\n": "<p></p>\n",
"<p> <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> are the tensors that store collection of <em>query</em>, <em>key</em> and <em>value</em> vectors. They have shape <span translate=no>_^_3_^_</span>.</p>\n<p><span translate=no>_^_4_^_</span> has shape <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> indicates whether for batch <span translate=no>_^_7_^_</span>, query at position <span translate=no>_^_8_^_</span> has access to key-value at position <span translate=no>_^_9_^_</span>.</p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u548c<span translate=no>_^_2_^_</span>\u662f\u5b58\u50a8<em>\u67e5\u8be2</em>\u3001<em>\u952e</em>\u548c<em>\u503c</em>\u5411\u91cf\u96c6\u5408\u7684\u5f20\u91cf\u3002\u5b83\u4eec\u6709\u5f62\u72b6<span translate=no>_^_3_^_</span>\u3002</p>\n<p><span translate=no>_^_4_^_</span>\u6709\u5f62\u72b6<span translate=no>_^_5_^_</span>\u5e76<span translate=no>_^_6_^_</span>\u6307\u793a\u662f\u5426\u4e3a\u6279\u91cf\u67e5\u8be2<span translate=no>_^_7_^_</span>\uff0c\u4f4d\u7f6e\u5904\u7684\u67e5\u8be2<span translate=no>_^_8_^_</span>\u6709\u6743\u8bbf\u95ee\u4f4d\u7f6e\u5904\u7684\u952e\u503c<span translate=no>_^_9_^_</span>\u3002</p>\n",
"<p> Simple test function to see the slopes.</p>\n": "<p>\u67e5\u770b\u659c\u7387\u7684\u7b80\u5355\u6d4b\u8bd5\u529f\u80fd\u3002</p>\n",
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
"<p><span translate=no>_^_0_^_</span> Note that we take steps by <span translate=no>_^_1_^_</span> to avoid slopes added previously. </p>\n": "<p><span translate=no>_^_0_^_</span>\u8bf7\u6ce8\u610f\uff0c\u6211\u4eec\u4f1a\u91c7\u53d6\u63aa\u65bd<span translate=no>_^_1_^_</span>\u907f\u514d\u4e4b\u524d\u6dfb\u52a0\u7684\u659c\u5761\u3002</p>\n",
"<p><span translate=no>_^_0_^_</span> attention along the key sequence dimension <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u5173\u6ce8\u6309\u952e\u5e8f\u5217\u7ef4\u5ea6<span translate=no>_^_1_^_</span></p>\n",
"<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> has shape <span translate=no>_^_1_^_</span> </p>\n",
"<p><span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> have shape <span translate=no>_^_3_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\uff0c<span translate=no>_^_1_^_</span>\u5e76\u4e14<span translate=no>_^_2_^_</span>\u6709\u5f62\u72b6<span translate=no>_^_3_^_</span></p>\n",
"<p>ALiBi only works with causal masks. </p>\n": "<p>AliBi \u4ec5\u9002\u7528\u4e8e\u56e0\u679c\u53e3\u7f69\u3002</p>\n",
"<p>Add AliBi biases to attention scores. ALiBi biases has shape <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> has shape <span translate=no>_^_2_^_</span> </p>\n": "<p>\u5c06 AliBi \u504f\u89c1\u6dfb\u52a0\u5230\u6ce8\u610f\u529b\u5206\u6570\u4e2d\u3002AliBi \u504f\u89c1\u6709\u5f62<span translate=no>_^_0_^_</span>\u72b6<span translate=no>_^_1_^_</span>\u4e5f\u6709\u5f62\u72b6<span translate=no>_^_2_^_</span></p>\n",
"<p>Add head dimension to mask and check its shape. </p>\n": "<p>\u5c06\u5934\u90e8\u5c3a\u5bf8\u6dfb\u52a0\u5230\u8499\u7248\u5e76\u68c0\u67e5\u5176\u5f62\u72b6\u3002</p>\n",
"<p>Apply dropout </p>\n": "<p>\u7533\u8bf7\u9000\u5b66</p>\n",
"<p>Apply mask </p>\n": "<p>\u6d82\u62b9\u9762\u819c</p>\n",
"<p>Calculate distances <span translate=no>_^_0_^_</span> Here we calculate the distances using the mask.</p>\n<p>Since it&#x27;s causal mask we can just use <span translate=no>_^_1_^_</span> too. <span translate=no>_^_2_^_</span> </p>\n": "<p>\u8ba1\u7b97\u8ddd\u79bb<span translate=no>_^_0_^_</span>\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528\u63a9\u7801\u8ba1\u7b97\u8ddd\u79bb\u3002</p>\n<p>\u65e2\u7136\u5b83\u662f\u56e0\u679c\u63a9\u7801\uff0c\u6211\u4eec<span translate=no>_^_1_^_</span>\u4e5f\u53ef\u4ee5\u4f7f\u7528\u3002<span translate=no>_^_2_^_</span></p>\n",
"<p>Compute attention scores <span translate=no>_^_0_^_</span>. This gives a tensor of shape <span translate=no>_^_1_^_</span>. </p>\n": "<p>\u8ba1\u7b97\u6ce8\u610f\u529b\u5206\u6570<span translate=no>_^_0_^_</span>\u3002\u8fd9\u7ed9\u51fa\u4e86\u5f62\u72b6\u7684\u5f20\u91cf<span translate=no>_^_1_^_</span>\u3002</p>\n",
"<p>Concatenate multiple heads </p>\n": "<p>\u8fde\u63a5\u591a\u4e2a\u5934</p>\n",
"<p>Concatenate the slopes with the remaining slopes. </p>\n": "<p>\u5c06\u659c\u5761\u4e0e\u5176\u4f59\u7684\u659c\u5761\u8fde\u63a5\u8d77\u6765\u3002</p>\n",
"<p>Create AliBi biases if it&#x27;s not cached </p>\n": "<p>\u5982\u679c AliBI \u672a\u88ab\u7f13\u5b58\uff0c\u5219\u521b\u5efa\u504f\u5dee</p>\n",
"<p>Get slopes <span translate=no>_^_0_^_</span> for each head </p>\n": "<p>\u83b7\u53d6\u6bcf\u4e2a<span translate=no>_^_0_^_</span>\u5934\u90e8\u7684\u659c\u7387</p>\n",
"<p>Get the closest power of 2 to <span translate=no>_^_0_^_</span>. If <span translate=no>_^_1_^_</span> is not a power of 2, then we first calculate slopes to the closest (smaller) power of 2, and then add the remaining slopes. </p>\n": "<p>\u83b7\u5f97\u6700\u63a5\u8fd1 2 \u7684\u5e42<span translate=no>_^_0_^_</span>\u3002\u5982\u679c\u4e0d<span translate=no>_^_1_^_</span>\u662f 2 \u7684\u5e42\uff0c\u90a3\u4e48\u6211\u4eec\u9996\u5148\u8ba1\u7b97\u659c\u7387\u5230\u6700\u63a5\u8fd1\uff08\u8f83\u5c0f\uff09\u7684 2 \u5e42\uff0c\u7136\u540e\u518d\u52a0\u4e0a\u5269\u4f59\u7684\u659c\u7387\u3002</p>\n",
"<p>If <span translate=no>_^_0_^_</span> is not a power of 2, then we add the remaining slopes. We calculate the remaining slopes for <span translate=no>_^_1_^_</span> (avoiding slopes added previously). And pick the slopes upto <span translate=no>_^_2_^_</span>. </p>\n": "<p>\u5982\u679c\u4e0d<span translate=no>_^_0_^_</span>\u662f 2 \u7684\u5e42\uff0c\u90a3\u4e48\u6211\u4eec\u5c06\u5269\u4f59\u7684\u659c\u7387\u76f8\u52a0\u3002\u6211\u4eec\u8ba1\u7b97\u5269\u4f59\u7684\u659c\u7387<span translate=no>_^_1_^_</span>\uff08\u907f\u514d\u4e4b\u524d\u6dfb\u52a0\u7684\u659c\u7387\uff09\u3002\u7136\u540e\u9009\u62e9\u659c\u5761<span translate=no>_^_2_^_</span>\u3002</p>\n",
"<p>Multiply by values <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4e58\u4ee5\u503c<span translate=no>_^_0_^_</span></p>\n",
"<p>Multiply them pair-wise to get the AliBi bias matrix </p>\n": "<p>\u5c06\u5b83\u4eec\u6210\u5bf9\u4e58\u4ee5\u5f97\u5230 AliBi \u504f\u5dee\u77e9\u9635</p>\n",
"<p>Output layer </p>\n": "<p>\u8f93\u51fa\u5c42</p>\n",
"<p>Prepare <span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> and <span translate=no>_^_2_^_</span> for attention computation. These will then have shape <span translate=no>_^_3_^_</span>. </p>\n": "<p>\u51c6\u5907<span translate=no>_^_0_^_</span>\uff0c<span translate=no>_^_1_^_</span>\u5e76<span translate=no>_^_2_^_</span>\u8fdb\u884c\u6ce8\u610f\u529b\u8ba1\u7b97\u3002\u7136\u540e\u8fd9\u4e9b\u5c31\u4f1a\u6709\u5f62\u72b6<span translate=no>_^_3_^_</span>\u3002</p>\n",
"<p>Scale scores <span translate=no>_^_0_^_</span> </p>\n": "<p>\u97f3\u9636\u5206\u6570<span translate=no>_^_0_^_</span></p>\n",
"<p>To cache AliBi the biases </p>\n": "<p>\u7f13\u5b58 AliBi \u7684\u504f\u89c1</p>\n",
"Attention with Linear Biases (ALiBi)": "\u6ce8\u610f\u7ebf\u6027\u504f\u5dee (AliBI)",
"Documented implementation with explanations of Attention with Linear Biases (ALiBi)": "\u8bb0\u5f55\u5b9e\u73b0\uff0c\u5e76\u89e3\u91ca\u7ebf\u6027\u504f\u5dee\u6ce8\u610f\u529b (AliBi)"
}
translate_cache/transformers/alibi/experiment.ja.json
+38
View File
@@ -0,0 +1,38 @@
{
"<h1><a href=\"index.html\">Attention with Linear Biases (ALiBi)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">ALiBi model</a>.</p>\n<p>This is based on <a href=\"../gpt/index.html\">our GPT model</a>.</p>\n": "<h1><a href=\"index.html\">\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9 (AliBi) \u5b9f\u9a13\u306b\u3088\u308b\u6ce8\u610f</a></h1>\n<p><a href=\"index.html\">\u3053\u308c\u306f\u3001AliBi \u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u306e PyTorch \u5b9f\u9a13\u3067\u3059\u3002</a></p>\n<p><a href=\"../gpt/index.html\">\u3053\u308c\u306f\u5f53\u793e\u306eGPT\u30e2\u30c7\u30eb\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059</a>\u3002</p>\n",
"<h2>Configurations</h2>\n<p>We extend <a href=\"../gpt/index.html\">GPT configurations</a> and change the attention mechanism.</p>\n": "<h2>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h2>\n<p><a href=\"../gpt/index.html\">GPT\u69cb\u6210\u3092\u62e1\u5f35\u3057</a>\u3001\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e1\u30ab\u30cb\u30ba\u30e0\u3092\u5909\u66f4\u3057\u307e\u3059\u3002</p>\n",
"<h3>ALiBi based Transformer configurations</h3>\n": "<h3>AliBi \u30d9\u30fc\u30b9\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc\u69cb\u6210</h3>\n",
"<p> </p>\n": "<p></p>\n",
"<p> Create an ALiBi attention module</p>\n": "<p>AliBi \u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u4f5c\u6210\u3059\u308b</p>\n",
"<p> Log losses at the initial and final tokens</p>\n": "<p>\u6700\u521d\u306e\u30c8\u30fc\u30af\u30f3\u3068\u6700\u5f8c\u306e\u30c8\u30fc\u30af\u30f3\u3067\u640d\u5931\u3092\u8a18\u9332\u3059\u308b</p>\n",
"<p> Shuffled validation data loader with <span translate=no>_^_0_^_</span> sequence length</p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b9\u9577\u306e\u30b7\u30e3\u30c3\u30d5\u30eb\u691c\u8a3c\u30c7\u30fc\u30bf\u30ed\u30fc\u30c0\u30fc <span translate=no>_^_0_^_</span></p>\n",
"<p>&#x27;text&#x27;: &#x27;tiny_shakespeare_no_split&#x27;, </p>\n": "<p>'\u30c6\u30ad\u30b9\u30c8': 'tiny_shakespeare_no_split'\u3001</p>\n",
"<p>ALiBi based transformer (defined below) </p>\n": "<p>ALiBi \u30d9\u30fc\u30b9\u306e\u30c8\u30e9\u30f3\u30b9\u30d5\u30a9\u30fc\u30de\u30fc (\u4ee5\u4e0b\u306b\u5b9a\u7fa9)</p>\n",
"<p>ALiBi doesn&#x27;t use positional embeddings </p>\n": "<p>AliBi \u306f\u4f4d\u7f6e\u57cb\u3081\u8fbc\u307f\u3092\u4f7f\u7528\u3057\u307e\u305b\u3093</p>\n",
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba <span translate=no>_^_0_^_</span></p>\n",
"<p>Create configs </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30b0\u306e\u4f5c\u6210</p>\n",
"<p>Create experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT \u306f GELU \u30a2\u30af\u30c6\u30a3\u30d9\u30fc\u30b7\u30e7\u30f3\u3092\u4f7f\u7528\u3057\u3066\u4f4d\u7f6e\u3054\u3068\u306e\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u3092\u884c\u3044\u307e\u3059</p>\n",
"<p>If there are more tokens that the training sequence length (during validation), </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3088\u308a\u3082\u591a\u304f\u306e\u30c8\u30fc\u30af\u30f3\u304c\u3042\u308b\u5834\u5408 (\u691c\u8a3c\u4e2d)\u3001</p>\n",
"<p>Log the loss at the final token </p>\n": "<p>\u6700\u5f8c\u306e\u30c8\u30fc\u30af\u30f3\u3067\u640d\u5931\u3092\u8a18\u9332\u3059\u308b</p>\n",
"<p>Log the loss at the first token </p>\n": "<p>\u6700\u521d\u306e\u30c8\u30fc\u30af\u30f3\u3067\u640d\u5931\u3092\u8a18\u9332\u3059\u308b</p>\n",
"<p>Log the loss at training sequence length </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30b7\u30fc\u30b1\u30f3\u30b9\u306e\u9577\u3055\u3067\u640d\u5931\u3092\u8a18\u9332\u3057\u307e\u3059</p>\n",
"<p>Longer validation set </p>\n": "<p>\u3088\u308a\u9577\u3044\u691c\u8a3c\u30bb\u30c3\u30c8</p>\n",
"<p>Override configurations </p>\n": "<p>\u30aa\u30fc\u30d0\u30fc\u30e9\u30a4\u30c9\u8a2d\u5b9a</p>\n",
"<p>Prompt separator is blank </p>\n": "<p>\u30d7\u30ed\u30f3\u30d7\u30c8\u30bb\u30d1\u30ec\u30fc\u30bf\u304c\u7a7a\u767d</p>\n",
"<p>Run training </p>\n": "<p>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c</p>\n",
"<p>Set all attention mechanisms to ALiBi </p>\n": "<p>\u3059\u3079\u3066\u306e\u6ce8\u610f\u30e1\u30ab\u30cb\u30ba\u30e0\u3092AliBi\u306b\u8a2d\u5b9a</p>\n",
"<p>Set models for saving and loading </p>\n": "<p>\u4fdd\u5b58\u304a\u3088\u3073\u8aad\u307f\u8fbc\u307f\u7528\u306e\u30e2\u30c7\u30eb\u3092\u8a2d\u5b9a\u3059\u308b</p>\n",
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u57cb\u3081\u8fbc\u307f\u3084\u30ed\u30b8\u30c3\u30c8\u306e\u751f\u6210\u306b\u4f7f\u7528\u3059\u308b\u30dc\u30ad\u30e3\u30d6\u30e9\u30ea\u30fc\u30b5\u30a4\u30ba\u3092\u8a2d\u5b9a</p>\n",
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
"<p>Starting prompt for sampling </p>\n": "<p>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306e\u958b\u59cb\u30d7\u30ed\u30f3\u30d7\u30c8</p>\n",
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u3054\u3068\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3068\u691c\u8a3c\u3092\u5207\u308a\u66ff\u3048\u308b <span translate=no>_^_0_^_</span></p>\n",
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span>\u6642\u4ee3\u306b\u5408\u308f\u305b\u305f\u5217\u8eca</p>\n",
"<p>Transformer configurations </p>\n": "<p>\u5909\u5727\u5668\u69cb\u6210</p>\n",
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u30bf\u30a4\u30cb\u30fc\u30fb\u30b7\u30a7\u30a4\u30af\u30b9\u30d4\u30a2\u30fb\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u3092\u4f7f\u3046</p>\n",
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b3\u30f3\u30c6\u30ad\u30b9\u30c8\u30b5\u30a4\u30ba\u3092\u6b21\u306e\u5024\u306b\u3057\u3066\u304f\u3060\u3055\u3044 <span translate=no>_^_0_^_</span></p>\n",
"<p>Use character level tokenizer </p>\n": "<p>\u30ad\u30e3\u30e9\u30af\u30bf\u30fc\u30ec\u30d9\u30eb\u306e\u30c8\u30fc\u30af\u30ca\u30a4\u30b6\u30fc\u3092\u4f7f\u3046</p>\n",
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p><a href=\"../configs.html#TransformerConfigs\">\u8a2d\u5b9a\u53ef\u80fd\u306a\u30c8\u30e9\u30f3\u30b9\u5b9f\u88c5\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\n",
"Attention with Linear Biases (ALiBi) Experiment": "\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9 (AliBi) \u5b9f\u9a13\u306b\u3088\u308b\u6ce8\u610f",
"This experiment trains an Attention with Linear Biases (ALiBi) based model on Tiny Shakespeare dataset.": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001Tiny Shakespeare\u306e\u30c7\u30fc\u30bf\u30bb\u30c3\u30c8\u306b\u57fa\u3065\u3044\u3066\u3001\u7dda\u5f62\u30d0\u30a4\u30a2\u30b9\uff08AliBi\uff09\u306b\u57fa\u3065\u304f\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
}
translate_cache/transformers/alibi/experiment.si.json
+38
View File
@@ -0,0 +1,38 @@
{
"<h1><a href=\"index.html\">Attention with Linear Biases (ALiBi)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">ALiBi model</a>.</p>\n<p>This is based on <a href=\"../gpt/index.html\">our GPT model</a>.</p>\n<p><a href=\"https://app.labml.ai/run/1454f9ba044a11ed8364e5e321a405ac\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1><a href=\"index.html\">\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 (\u0d85\u0dbd\u0dd2\u0db6\u0dd3) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4</a> \u0d9a\u0dbb\u0db1\u0dca\u0db1</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"index.html\">\u0d85\u0dbd\u0dd2\u0db6\u0dd3 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca</a>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0db4\u0dba\u0dd2\u0da7\u0ddd\u0da0\u0dca \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dd2. </p>\n<p>\u0db8\u0dd9\u0dba <a href=\"../gpt/index.html\">\u0d85\u0db4\u0d9c\u0dda GPT \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</a>\u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dda. </p>\n<p><a href=\"https://app.labml.ai/run/1454f9ba044a11ed8364e5e321a405ac\"><span translate=no>_^_0_^_</span></a></p>\n",
"<h2>Configurations</h2>\n<p>We extend <a href=\"../gpt/index.html\">GPT configurations</a> and change the attention mechanism.</p>\n": "<h2>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h2>\n<p>\u0d85\u0db4\u0dd2 <a href=\"../gpt/index.html\">GPT \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a> \u0daf\u0dd3\u0dbb\u0dca extend \u0d9a\u0dbb \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dba\u0dcf\u0db1\u0dca\u0dad\u0dca\u0dbb\u0dab\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dbb\u0db8\u0dd4. </p>\n",
"<h3>ALiBi based Transformer configurations</h3>\n": "<h3>\u0d85\u0dbd\u0dd2\u0db6\u0dd3\u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</h3>\n",
"<p> </p>\n": "<p> </p>\n",
"<p> Create an ALiBi attention module</p>\n": "<p> \u0d85\u0dbd\u0dd2\u0db6\u0dd3\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0db8\u0ddc\u0da9\u0dd2\u0dba\u0dd4\u0dbd\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</p>\n",
"<p> Log losses at the initial and final tokens</p>\n": "<p> \u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a\u0dc4\u0dcf \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0da7\u0ddd\u0d9a\u0db1 \u0dc0\u0dbd\u0daf\u0dd3 \u0db4\u0dcf\u0da9\u0dd4 \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n",
"<p> Shuffled validation data loader with <span translate=no>_^_0_^_</span> sequence length</p>\n": "<p> <span translate=no>_^_0_^_</span> \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0daf\u0dd2\u0d9c \u0dc3\u0dc4\u0dd2\u0dad \u0dc0\u0dbd\u0d82\u0d9c\u0dd4\u0d9a\u0dbb\u0dab \u0daf\u0dad\u0dca\u0dad \u0db4\u0dd0\u0da7\u0dc0\u0dd4\u0db8 \u0db8\u0dcf\u0dbb\u0dd4 \u0d9a\u0dbb \u0d87\u0dad</p>\n",
"<p>&#x27;text&#x27;: &#x27;tiny_shakespeare_no_split&#x27;, </p>\n": "<p>'text ':' \u0da7\u0dd2\u0db1\u0dd2_\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca_\u0db1\u0ddc_\u0db6\u0dd9\u0daf\u0dd3\u0db8\u0dca ', </p>\n",
"<p>ALiBi based transformer (defined below) </p>\n": "<p>\u0d85\u0dbd\u0dd2\u0db6\u0dd3\u0db4\u0daf\u0db1\u0db8\u0dca \u0d9a\u0dbb\u0d9c\u0dad\u0dca \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca (\u0db4\u0dc4\u0dad \u0d85\u0dbb\u0dca\u0dae \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d87\u0dad) </p>\n",
"<p>ALiBi doesn&#x27;t use positional embeddings </p>\n": "<p>AliBi\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0dd3\u0dba \u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0db1\u0ddc\u0d9a\u0dbb\u0dba\u0dd2 </p>\n",
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca\u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba <span translate=no>_^_0_^_</span> </p>\n",
"<p>Create configs </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
"<p>Create experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT\u0dc3\u0dca\u0dae\u0dcf\u0db1\u0d9c\u0dad \u0db1\u0dd0\u0dab\u0dc0\u0dad\u0dca \u0db4\u0ddd\u0dc2\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf GELU \u0dc3\u0d9a\u0dca\u0dbb\u0dd2\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0dba\u0dd2 </p>\n",
"<p>If there are more tokens that the training sequence length (during validation), </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda \u0daf\u0dd2\u0d9c (\u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda\u0daf\u0dd3) \u0dc0\u0dd0\u0da9\u0dd2 \u0da7\u0ddd\u0d9a\u0db1 \u0dad\u0dd2\u0db6\u0dda \u0db1\u0db8\u0dca, </p>\n",
"<p>Log the loss at the final token </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
"<p>Log the loss at the first token </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dda \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
"<p>Log the loss at training sequence length </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dba\u0dda \u0daf\u0dd2\u0d9c \u0daf\u0dd3 \u0d85\u0dbd\u0dcf\u0db7\u0dba \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
"<p>Longer validation set </p>\n": "<p>\u0daf\u0dd2\u0d9c\u0dd4\u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba </p>\n",
"<p>Override configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0d85\u0db7\u0dd2\u0db6\u0dc0\u0dcf \u0dba\u0db1\u0dca\u0db1 </p>\n",
"<p>Prompt separator is blank </p>\n": "<p>\u0d9a\u0da9\u0dd2\u0db1\u0db8\u0dca\u0db6\u0dd9\u0daf\u0dd4\u0db8\u0dca\u0d9a\u0dbb\u0dd4 \u0dc4\u0dd2\u0dc3\u0dca \u0dba </p>\n",
"<p>Run training </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0db0\u0dcf\u0dc0\u0db1\u0dba </p>\n",
"<p>Set all attention mechanisms to ALiBi </p>\n": "<p>\u0dc3\u0dd2\u0dba\u0dc5\u0dd4\u0db8\u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dda \u0dba\u0dcf\u0db1\u0dca\u0dad\u0dca\u0dbb\u0dab \u0d85\u0dbd\u0dd2\u0db6\u0dd3 \u0dc0\u0dd9\u0dad \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
"<p>Set models for saving and loading </p>\n": "<p>\u0d89\u0dad\u0dd2\u0dbb\u0dd2\u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0dc4 \u0db4\u0dd0\u0da7\u0dc0\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2 \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u0d9a\u0dcf\u0dc0\u0dd0\u0daf\u0dca\u0daf\u0dd3\u0db8\u0dca\u0dc3\u0dc4 \u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0d8b\u0dad\u0dca\u0db4\u0dcf\u0daf\u0db1\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0da0\u0db1 \u0db8\u0dcf\u0dbd\u0dcf\u0dc0 \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
"<p>Starting prompt for sampling </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd3\u0db8\u0dc3\u0db3\u0dc4\u0dcf \u0dc0\u0dd2\u0db8\u0dc3\u0dd4\u0db8\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u0d91\u0d9a\u0dca <span translate=no>_^_0_^_</span> \u0dba\u0dd4\u0d9c\u0dba\u0d9a\u0da7 \u0dc0\u0dbb\u0d9a\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0dc4 \u0dc0\u0dbd\u0d82\u0d9c\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0d85\u0dad\u0dbb \u0db8\u0dcf\u0dbb\u0dd4 \u0dc0\u0db1\u0dca\u0db1 </p>\n",
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p><span translate=no>_^_0_^_</span> Epochs \u0dc3\u0db3\u0dc4\u0dcf \u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
"<p>Transformer configurations </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca </p>\n",
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d9a\u0dc3\u0db1\u0dca\u0daf\u0dbb\u0dca\u0db7\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf <span translate=no>_^_0_^_</span> </p>\n",
"<p>Use character level tokenizer </p>\n": "<p>\u0d85\u0d9a\u0dca\u0dc2\u0dbb\u0db8\u0da7\u0dca\u0da7\u0db8\u0dda \u0da7\u0ddd\u0d9a\u0db1\u0dba\u0dd2\u0dc3\u0dbb\u0dca \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u0d85\u0db4\u0d9c\u0dda <a href=\"../configs.html#TransformerConfigs\">\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9c\u0dad \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0da7\u0dca\u0dbb\u0dcf\u0db1\u0dca\u0dc3\u0dca\u0dc6\u0ddd\u0db8\u0dbb\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db8\u0dd4 </p>\n",
"Attention with Linear Biases (ALiBi) Experiment": "\u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0db4\u0d9a\u0dca\u0dc2\u0d9c\u0dca\u0dbb\u0dcf\u0dc4\u0dd3 (\u0d85\u0dbd\u0dd2\u0db6\u0dd3) \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0db8\u0d9f \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1",
"This experiment trains an Attention with Linear Biases (ALiBi) based model on Tiny Shakespeare dataset.": "\u0db8\u0dd9\u0db8 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dd4\u0da9\u0dcf \u0dc2\u0dda\u0d9a\u0dca\u0dc3\u0dca\u0db4\u0dd2\u0dba\u0dbb\u0dca \u0daf\u0dad\u0dca\u0dad \u0d9a\u0da7\u0dca\u0da7\u0dbd\u0dba \u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca \u0dc0\u0dd6 \u0dbb\u0dda\u0d9b\u0dd3\u0dba \u0d85\u0d9c\u0dad\u0dd3\u0db1\u0dca (\u0d85\u0dbd\u0dd2\u0db6\u0dd3) \u0dc3\u0db8\u0d9f \u0d85\u0dc0\u0db0\u0dcf\u0db1\u0dba \u0dba\u0ddc\u0db8\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
}
translate_cache/transformers/alibi/experiment.zh.json
+38
View File
@@ -0,0 +1,38 @@
{
"<h1><a href=\"index.html\">Attention with Linear Biases (ALiBi)</a> Experiment</h1>\n<p>This is an annotated PyTorch experiment to train a <a href=\"index.html\">ALiBi model</a>.</p>\n<p>This is based on <a href=\"../gpt/index.html\">our GPT model</a>.</p>\n": "<h1><a href=\"index.html\">\u7ebf\u6027\u504f\u5dee\uff08AliBI\uff09\u5b9e\u9a8c\u4e2d\u7684\u6ce8\u610f\u529b</a></h1>\n<p>\u8fd9\u662f\u4e00\u9879\u5e26\u6ce8\u91ca\u7684 PyTorch \u5b9e\u9a8c\uff0c\u7528\u4e8e\u8bad\u7ec3 A <a href=\"index.html\">liBI \u6a21\u578b</a>\u3002</p>\n<p>\u8fd9\u662f\u57fa\u4e8e<a href=\"../gpt/index.html\">\u6211\u4eec\u7684 GPT \u6a21\u578b</a>\u3002</p>\n",
"<h2>Configurations</h2>\n<p>We extend <a href=\"../gpt/index.html\">GPT configurations</a> and change the attention mechanism.</p>\n": "<h2>\u914d\u7f6e</h2>\n<p>\u6211\u4eec\u6269\u5c55<a href=\"../gpt/index.html\">\u4e86 GPT \u914d\u7f6e</a>\u5e76\u66f4\u6539\u4e86\u6ce8\u610f\u673a\u5236\u3002</p>\n",
"<h3>ALiBi based Transformer configurations</h3>\n": "<h3>\u57fa\u4e8e AliBI \u7684\u53d8\u538b\u5668\u914d\u7f6e</h3>\n",
"<p> </p>\n": "<p></p>\n",
"<p> Create an ALiBi attention module</p>\n": "<p>\u521b\u5efa\u4e00\u4e2a AliBI \u6ce8\u610f\u529b\u6a21\u5757</p>\n",
"<p> Log losses at the initial and final tokens</p>\n": "<p>\u8bb0\u5f55\u521d\u59cb\u548c\u6700\u7ec8\u4ee3\u5e01\u7684\u635f\u5931</p>\n",
"<p> Shuffled validation data loader with <span translate=no>_^_0_^_</span> sequence length</p>\n": "<p>\u4f7f\u7528<span translate=no>_^_0_^_</span>\u5e8f\u5217\u957f\u5ea6\u6539\u7ec4\u9a8c\u8bc1\u6570\u636e\u52a0\u8f7d\u5668</p>\n",
"<p>&#x27;text&#x27;: &#x27;tiny_shakespeare_no_split&#x27;, </p>\n": "<p>'text': 'tiny_shakespeare_no_split '\uff0c</p>\n",
"<p>ALiBi based transformer (defined below) </p>\n": "<p>\u57fa\u4e8e AliBI \u7684\u8f6c\u6362\u5668\uff08\u5b9a\u4e49\u89c1\u4e0b\u6587\uff09</p>\n",
"<p>ALiBi doesn&#x27;t use positional embeddings </p>\n": "<p>AliBI \u4e0d\u4f7f\u7528\u4f4d\u7f6e\u5d4c\u5165</p>\n",
"<p>Batch size <span translate=no>_^_0_^_</span> </p>\n": "<p>\u6279\u91cf\u5927\u5c0f<span translate=no>_^_0_^_</span></p>\n",
"<p>Create configs </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
"<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
"<p>GPT uses GELU activation for position wise feedforward </p>\n": "<p>GPT \u4f7f\u7528 GELU \u6fc0\u6d3b\u8fdb\u884c\u4f4d\u7f6e\u660e\u667a\u524d\u9988</p>\n",
"<p>If there are more tokens that the training sequence length (during validation), </p>\n": "<p>\u5982\u679c\u8bad\u7ec3\u5e8f\u5217\u957f\u5ea6\uff08\u5728\u9a8c\u8bc1\u671f\u95f4\uff09\u6709\u66f4\u591a\u7684\u4ee4\u724c\uff0c</p>\n",
"<p>Log the loss at the final token </p>\n": "<p>\u8bb0\u5f55\u6700\u7ec8\u4ee3\u5e01\u7684\u635f\u5931</p>\n",
"<p>Log the loss at the first token </p>\n": "<p>\u8bb0\u5f55\u7b2c\u4e00\u4e2a\u4ee4\u724c\u7684\u635f\u5931</p>\n",
"<p>Log the loss at training sequence length </p>\n": "<p>\u8bb0\u5f55\u8bad\u7ec3\u5e8f\u5217\u957f\u5ea6\u7684\u635f\u5931</p>\n",
"<p>Longer validation set </p>\n": "<p>\u66f4\u957f\u7684\u9a8c\u8bc1\u96c6</p>\n",
"<p>Override configurations </p>\n": "<p>\u8986\u76d6\u914d\u7f6e</p>\n",
"<p>Prompt separator is blank </p>\n": "<p>\u63d0\u793a\u5206\u9694\u7b26\u4e3a\u7a7a</p>\n",
"<p>Run training </p>\n": "<p>\u8dd1\u6b65\u8bad\u7ec3</p>\n",
"<p>Set all attention mechanisms to ALiBi </p>\n": "<p>\u5c06\u6240\u6709\u5173\u6ce8\u673a\u5236\u8bbe\u7f6e\u4e3a AliBI</p>\n",
"<p>Set models for saving and loading </p>\n": "<p>\u8bbe\u7f6e\u7528\u4e8e\u4fdd\u5b58\u548c\u52a0\u8f7d\u7684\u6a21\u578b</p>\n",
"<p>Set the vocabulary sizes for embeddings and generating logits </p>\n": "<p>\u8bbe\u7f6e\u5d4c\u5165\u548c\u751f\u6210 logit \u7684\u8bcd\u6c47\u91cf\u5927\u5c0f</p>\n",
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
"<p>Starting prompt for sampling </p>\n": "<p>\u5f00\u59cb\u91c7\u6837\u63d0\u793a</p>\n",
"<p>Switch between training and validation for <span translate=no>_^_0_^_</span> times per epoch </p>\n": "<p>\u5728\u8bad\u7ec3\u548c\u9a8c\u8bc1\u4e4b\u95f4\u5207\u6362\u6bcf\u4e2a\u7eaa\u5143\u7684<span translate=no>_^_0_^_</span>\u6b21\u6570</p>\n",
"<p>Train for <span translate=no>_^_0_^_</span> epochs </p>\n": "<p>\u4e3a<span translate=no>_^_0_^_</span>\u65f6\u4ee3\u800c\u8bad\u7ec3</p>\n",
"<p>Transformer configurations </p>\n": "<p>\u53d8\u538b\u5668\u914d\u7f6e</p>\n",
"<p>Use Tiny Shakespeare dataset </p>\n": "<p>\u4f7f\u7528\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</p>\n",
"<p>Use a context size of <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4f7f\u7528\u4e0a\u4e0b\u6587\u5927\u5c0f\u4e3a<span translate=no>_^_0_^_</span></p>\n",
"<p>Use character level tokenizer </p>\n": "<p>\u4f7f\u7528\u89d2\u8272\u7b49\u7ea7\u5206\u8bcd\u5668</p>\n",
"<p>We use our <a href=\"../configs.html#TransformerConfigs\">configurable transformer implementation</a> </p>\n": "<p>\u6211\u4eec\u4f7f\u7528\u6211\u4eec\u7684<a href=\"../configs.html#TransformerConfigs\">\u53ef\u914d\u7f6e\u53d8\u538b\u5668\u5b9e\u73b0</a></p>\n",
"Attention with Linear Biases (ALiBi) Experiment": "\u6ce8\u610f\u7ebf\u6027\u504f\u5dee (AliBi) \u5b9e\u9a8c",
"This experiment trains an Attention with Linear Biases (ALiBi) based model on Tiny Shakespeare dataset.": "\u672c\u5b9e\u9a8c\u5728 Tiny Shakespeare \u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u57fa\u4e8e\u7ebf\u6027\u504f\u5dee\u6ce8\u610f\u529b\uff08AliBi\uff09\u7684\u6a21\u578b\u3002"
}