{
 "<h1>Fine Tune GPT-NeoX</h1>\n<p>This shows how to fine tune GPT-NeoX with pipeline parallelism.</p>\n": "<h1>Fine Tune GPT-NEOX</h1>\n<p>\u8fd9\u8bf4\u660e\u4e86\u5982\u4f55\u5229\u7528\u7ba1\u9053\u5e76\u884c\u5ea6\u5bf9 GPT-NEOX \u8fdb\u884c\u5fae\u8c03\u3002</p>\n",
 "<h3>Create fine tuner for biases</h3>\n": "<h3>\u4e3a\u504f\u89c1\u521b\u5efa\u7cbe\u7ec6\u7684\u8c03\u8c10\u5668</h3>\n",
 "<h3>Create pipeline parallel model</h3>\n": "<h3>\u521b\u5efa\u7ba1\u9053\u5e76\u884c\u6a21\u578b</h3>\n",
 "<h3>Load GPT-NeoX layers</h3>\n": "<h3>\u52a0\u8f7d GPT-NEOX \u56fe\u5c42</h3>\n",
 "<h4>Tiny Shakespeare dataset</h4>\n": "<h4>\u5c0f\u838e\u58eb\u6bd4\u4e9a\u6570\u636e\u96c6</h4>\n",
 "<p> </p>\n": "<p></p>\n",
 "<p>Create Fairscale Pipe module </p>\n": "<p>\u521b\u5efa\u516c\u5e73\u89c4\u6a21\u7ba1\u9053\u6a21\u5757</p>\n",
 "<p>Create experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
 "<p>Create the Pipe module </p>\n": "<p>\u521b\u5efa\u7ba1\u9053\u6a21\u5757</p>\n",
 "<p>Devices for each GPU </p>\n": "<p>\u6bcf\u4e2a GPU \u7684\u8bbe\u5907</p>\n",
 "<p>Get the layer distribution across GPUs </p>\n": "<p>\u83b7\u53d6\u8de8 GPU \u7684\u5c42\u5206\u5e03</p>\n",
 "<p>Initialize configs </p>\n": "<p>\u521d\u59cb\u5316\u914d\u7f6e</p>\n",
 "<p>Initialize the model. Do this before the loop for cleaner logs. </p>\n": "<p>\u521d\u59cb\u5316\u6a21\u578b\u3002\u5728\u5faa\u73af\u4e4b\u524d\u6267\u884c\u6b64\u64cd\u4f5c\u4ee5\u83b7\u5f97\u66f4\u6e05\u6670\u7684\u65e5\u5fd7\u3002</p>\n",
 "<p>Make sure the finetuner is initialized </p>\n": "<p>\u786e\u4fdd\u5fae\u8c03\u5668\u5df2\u521d\u59cb\u5316</p>\n",
 "<p>Mark biases as trainable </p>\n": "<p>\u5c06\u504f\u89c1\u6807\u8bb0\u4e3a\u53ef\u8bad\u7ec3</p>\n",
 "<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
 "<p>Train </p>\n": "<p>\u706b\u8f66</p>\n",
 "Fine Tune GPT-NeoX": "Fine Tune GPT-NEOX",
 "Fine tune GPT-NeoX biases with Fairscale pipeline parallel module": "\u4f7f\u7528 Fairscale \u7ba1\u9053\u5e76\u884c\u6a21\u5757\u5fae\u8c03 GPT-NEOX \u504f\u5dee"
}