chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,131 @@
|
||||
{
|
||||
"<h1>Zero-DP Memory Optimization</h1>\n<p>This is an implementation of Zero-DP introduced in the paper <a href=\"https://arxiv.org/abs/1910.02054\">ZeRO: Memory Optimization Towards Training A Trillion Parameter Models</a>,</p>\n<p>It keeps shards of the optimizer state, gradients and parameters into multiple devices/nodes. It reduces the memory consumption to <span translate=no>_^_0_^_</span> of the original model, where <span translate=no>_^_1_^_</span> is the number of parameters, <span translate=no>_^_2_^_</span> is the number of shards, and <span translate=no>_^_3_^_</span> is number of optimizer bytes per parameter. <span translate=no>_^_4_^_</span> are the parameter and gradient memory assuming 16-bit precision; i.e. 2 bytes per parameter and gradient. <span translate=no>_^_5_^_</span> for Adam optimizer because it maintains a copy of parameters, and two moments per parameter in fp32.</p>\n<p>The communication volume of Zero-DP is <span translate=no>_^_6_^_</span>. For comparison data-parallel training has a communication volume of <span translate=no>_^_7_^_</span>.</p>\n<p>Although this is named <span translate=no>_^_8_^_</span>, we have only implemented the Zero-DP part of it and not the Zero-R memory optimizations which target residual memory consumption. Out implementation supports training only a subset of parameters.</p>\n<p>This implementation is inspired by <a href=\"https://fairscale.readthedocs.io/en/stable/api/nn/fsdp.html\">Fairscale FSDP</a>.</p>\n<p><a href=\"finetune_neox.html\">Here's a script to fine-tune</a> GPT NeoX using Zero-DP memory optimization.</p>\n": "<h1>\u96f6 DP \u5185\u5b58\u4f18\u5316</h1>\n<p>\u8fd9\u662f\u300a\u96f6<a href=\"https://arxiv.org/abs/1910.02054\">\uff1a\u8bad\u7ec3\u4e00\u4e07\u4ebf\u4e2a\u53c2\u6570\u6a21\u578b\u7684\u5185\u5b58\u4f18\u5316\u300b\u4e00\u6587\u4e2d\u4ecb\u7ecd\u7684\u96f6 DP \u7684\u5b9e\u73b0</a>\uff0c</p>\n<p>\u5b83\u5c06\u4f18\u5316\u5668\u72b6\u6001\u3001\u68af\u5ea6\u548c\u53c2\u6570\u7684\u5206\u7247\u4fdd\u5b58\u5230\u591a\u4e2a\u8bbe\u5907/\u8282\u70b9\u4e2d\u3002\u5b83\u51cf\u5c11\u4e86\u539f\u59cb\u6a21\u578b<span translate=no>_^_0_^_</span>\u7684\u5185\u5b58\u6d88\u8017\uff0c\u5176\u4e2d<span translate=no>_^_1_^_</span>\u662f\u53c2\u6570\u7684\u6570\u91cf\uff0c<span translate=no>_^_2_^_</span>\u662f\u5206\u7247\u7684\u6570\u91cf\uff0c<span translate=no>_^_3_^_</span>\u662f\u6bcf\u4e2a\u53c2\u6570\u7684\u4f18\u5316\u5668\u5b57\u8282\u6570\u3002<span translate=no>_^_4_^_</span>\u662f\u5047\u8bbe\u7cbe\u5ea6\u4e3a 16 \u4f4d\u7684\u53c2\u6570\u548c\u68af\u5ea6\u5b58\u50a8\u5668\uff1b\u5373\u6bcf\u4e2a\u53c2\u6570\u548c\u68af\u5ea6\u4e3a 2 \u4e2a\u5b57\u8282\u3002<span translate=no>_^_5_^_</span>\u5bf9\u4e8e Adam \u4f18\u5316\u5668\uff0c\u56e0\u4e3a\u5b83\u7ef4\u62a4\u53c2\u6570\u7684\u526f\u672c\uff0c\u5728 fp32 \u4e2d\u6bcf\u4e2a\u53c2\u6570\u4e24\u4e2a\u65f6\u523b\u3002</p>\n<p>\u96f6 DP \u7684\u901a\u4fe1\u91cf\u4e3a<span translate=no>_^_6_^_</span>\u3002\u6bd4\u8f83\u800c\u8a00\uff0c\u6570\u636e\u5e76\u884c\u8bad\u7ec3\u7684\u901a\u4fe1\u91cf\u4e3a<span translate=no>_^_7_^_</span>\u3002</p>\n<p>\u5c3d\u7ba1\u5b83\u88ab\u547d\u540d\u4e86<span translate=no>_^_8_^_</span>\uff0c\u4f46\u6211\u4eec\u53ea\u5b9e\u73b0\u4e86\u5176\u4e2d\u7684\u96f6 DP \u90e8\u5206\uff0c\u6ca1\u6709\u5b9e\u73b0\u9488\u5bf9\u5269\u4f59\u5185\u5b58\u6d88\u8017\u7684 Zero-R \u5185\u5b58\u4f18\u5316\u3002Out \u5b9e\u73b0\u4ec5\u652f\u6301\u8bad\u7ec3\u4e00\u90e8\u5206\u53c2\u6570\u3002</p>\n<p>\u6b64\u5b9e\u65bd\u7684\u7075\u611f\u6765\u81ea<a href=\"https://fairscale.readthedocs.io/en/stable/api/nn/fsdp.html\">\u516c\u5e73\u89c4\u6a21\u7684\u8d22\u52a1\u5b89\u5168\u53d1\u5c55\u8ba1\u5212</a>\u3002</p>\n<p><a href=\"finetune_neox.html\">\u4ee5\u4e0b\u662f\u4f7f\u7528\u96f6 DP \u5185\u5b58\u4f18\u5316\u5fae\u8c03 GPT NeoX \u7684\u811a\u672c</a>\u3002</p>\n",
|
||||
"<h2>Sequential module for <span translate=no>_^_0_^_</span> layers</h2>\n": "<h2><span translate=no>_^_0_^_</span>\u5c42\u7684\u987a\u5e8f\u6a21\u5757</h2>\n",
|
||||
"<h2>Zero3 Layer</h2>\n<p>Each layer of the model (or a combination of a few consecutive layers) should be wrapped in this module.</p>\n": "<h2>Zero3 \u5c42</h2>\n<p>\u6a21\u578b\u7684\u6bcf\u4e00\u5c42\uff08\u6216\u51e0\u4e2a\u8fde\u7eed\u5c42\u7684\u7ec4\u5408\uff09\u90fd\u5e94\u8be5\u5305\u88f9\u5728\u8fd9\u4e2a\u6a21\u5757\u4e2d\u3002</p>\n",
|
||||
"<h3>Backup the gradients of the current layer</h3>\n": "<h3>\u5907\u4efd\u5f53\u524d\u56fe\u5c42\u7684\u6e10\u53d8</h3>\n",
|
||||
"<h3>Fetch the parameters from all shards</h3>\n<p>This will fetch all the parameter data from all the nodes and rebuild the parameters on each node.</p>\n": "<h3>\u4ece\u6240\u6709\u5206\u7247\u4e2d\u83b7\u53d6\u53c2\u6570</h3>\n<p>\u8fd9\u5c06\u4ece\u6240\u6709\u8282\u70b9\u83b7\u53d6\u6240\u6709\u53c2\u6570\u6570\u636e\uff0c\u5e76\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u91cd\u5efa\u53c2\u6570\u3002</p>\n",
|
||||
"<h3>Forward pass</h3>\n": "<h3>\u5411\u524d\u4f20\u7403</h3>\n",
|
||||
"<h3>Get trainable chunk/shard of the parameters.</h3>\n<p>This is what we pass on to the optimizer on the current node.</p>\n": "<h3>\u83b7\u53d6\u53ef\u8bad\u7ec3\u7684\u53c2\u6570\u5757/\u5206\u7247\u3002</h3>\n<p>\u8fd9\u5c31\u662f\u6211\u4eec\u4f20\u9012\u7ed9\u5f53\u524d\u8282\u70b9\u4e0a\u7684\u4f18\u5316\u5668\u7684\u5185\u5bb9\u3002</p>\n",
|
||||
"<h4>Add backward hooks to the parameters of the current layer.</h4>\n": "<h4>\u5411\u5f53\u524d\u56fe\u5c42\u7684\u53c2\u6570\u6dfb\u52a0\u5411\u540e\u6302\u94a9\u3002</h4>\n",
|
||||
"<h4>Cleanup the parameter data</h4>\n<p>This will release all the memory used by the layer parameters.</p>\n": "<h4>\u6e05\u7406\u53c2\u6570\u6570\u636e</h4>\n<p>\u8fd9\u5c06\u91ca\u653e\u5c42\u53c2\u6570\u4f7f\u7528\u7684\u6240\u6709\u5185\u5b58\u3002</p>\n",
|
||||
"<h4>Create an empty tensor of the given shape.</h4>\n": "<h4>\u521b\u5efa\u7ed9\u5b9a\u5f62\u72b6\u7684\u7a7a\u5f20\u91cf\u3002</h4>\n",
|
||||
"<h4>Handle a backward event</h4>\n<p>This gets called by parameter backward hooks and the module backward hook.</p>\n": "<h4>\u5904\u7406\u5411\u540e\u4e8b\u4ef6</h4>\n<p>\u8fd9\u88ab\u53c2\u6570\u53cd\u5411\u94a9\u5b50\u548c\u6a21\u5757\u540e\u5411\u94a9\u5b50\u8c03\u7528\u3002</p>\n",
|
||||
"<h4>Merge all the parameters and pad it so that it's divisible by <span translate=no>_^_0_^_</span>.</h4>\n": "<h4>\u5408\u5e76\u6240\u6709\u53c2\u6570\u5e76\u586b\u5145\u5b83\uff0c\u4f7f\u5176\u53ef\u88ab\u6574\u9664<span translate=no>_^_0_^_</span>\u3002</h4>\n",
|
||||
"<h4>Module backward hook</h4>\n": "<h4>\u6a21\u5757\u5411\u540e\u6302\u94a9</h4>\n",
|
||||
"<h4>Parameter backward hook</h4>\n": "<h4>\u53c2\u6570\u5411\u540e\u6302\u94a9</h4>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Accumulate the gradients of each shard. It scatters the buffers across the nodes, and each node accumulates (reduces) the tensors it receives. </p>\n": "<p>\u7d2f\u79ef\u6bcf\u4e2a\u5206\u7247\u7684\u68af\u5ea6\u3002\u5b83\u5c06\u7f13\u51b2\u533a\u5206\u6563\u5230\u8282\u70b9\u4e0a\uff0c\u6bcf\u4e2a\u8282\u70b9\u7d2f\u79ef\uff08\u51cf\u5c11\uff09\u5b83\u6536\u5230\u7684\u5f20\u91cf\u3002</p>\n",
|
||||
"<p>Add a backward hook. This gets called when the gradients relative to the module are computed. </p>\n": "<p>\u6dfb\u52a0\u4e00\u4e2a\u5411\u540e\u94a9\u5b50\u3002\u5f53\u8ba1\u7b97\u76f8\u5bf9\u4e8e\u6a21\u5757\u7684\u68af\u5ea6\u65f6\uff0c\u4f1a\u8c03\u7528\u8be5\u51fd\u6570\u3002</p>\n",
|
||||
"<p>Add backward hooks to the parameters of the current layer if autograd is enabled. </p>\n": "\u5982\u679c@@ <p>\u542f\u7528\u4e86 autograd\uff0c\u5219\u5411\u5f53\u524d\u5c42\u7684\u53c2\u6570\u6dfb\u52a0\u5411\u540e\u6302\u94a9\u3002</p>\n",
|
||||
"<p>Add the backward hook </p>\n": "<p>\u6dfb\u52a0\u5411\u540e\u6302\u94a9</p>\n",
|
||||
"<p>All parameters should have the same type </p>\n": "<p>\u6240\u6709\u53c2\u6570\u90fd\u5e94\u5177\u6709\u76f8\u540c\u7684\u7c7b\u578b</p>\n",
|
||||
"<p>An empty tensor to receive the trainable and fixed parameters combined </p>\n": "<p>\u4e00\u4e2a\u7a7a\u5f20\u91cf\uff0c\u7528\u4e8e\u63a5\u6536\u53ef\u8bad\u7ec3\u53c2\u6570\u548c\u56fa\u5b9a\u53c2\u6570\u7684\u7ec4\u5408</p>\n",
|
||||
"<p>Assign the values from the continuous tensor </p>\n": "<p>\u4ece\u8fde\u7eed\u5f20\u91cf\u4e2d\u5206\u914d\u503c</p>\n",
|
||||
"<p>Broadcast the sizes </p>\n": "<p>\u5e7f\u64ad\u5c3a\u5bf8</p>\n",
|
||||
"<p>Buffer to store the gradients </p>\n": "<p>\u7528\u4e8e\u5b58\u50a8\u6e10\u53d8\u7684\u7f13\u51b2\u533a</p>\n",
|
||||
"<p>CUDA stream to back up (accumulate) gradients </p>\n": "<p>\u7528\u4e8e\u5907\u4efd\uff08\u7d2f\u79ef\uff09\u68af\u5ea6\u7684 CUDA \u6d41</p>\n",
|
||||
"<p>CUDA stream to backup/accumulate gradients </p>\n": "<p>\u7528\u4e8e\u5907\u4efd/\u7d2f\u79ef\u68af\u5ea6\u7684 CUDA \u6d41</p>\n",
|
||||
"<p>CUDA stream to featch parameters </p>\n": "<p>CUDA \u6d41\u5230\u7cbe\u9009\u53c2\u6570</p>\n",
|
||||
"<p>CUDA stream to fetch parameters </p>\n": "<p>\u7528\u4e8e\u83b7\u53d6\u53c2\u6570\u7684 CUDA \u6d41</p>\n",
|
||||
"<p>Calculate the chunk sizes of trainable and fixed params </p>\n": "<p>\u8ba1\u7b97\u53ef\u8bad\u7ec3\u53c2\u6570\u548c\u56fa\u5b9a\u53c2\u6570\u7684\u533a\u5757\u5927\u5c0f</p>\n",
|
||||
"<p>Change the storage size of the parameter. This was set to <span translate=no>_^_0_^_</span> when we cleaned up the parameters. </p>\n": "<p>\u66f4\u6539\u53c2\u6570\u7684\u5b58\u50a8\u5927\u5c0f\u3002\u8fd9\u662f\u6211\u4eec\u6e05\u7406\u53c2\u6570<span translate=no>_^_0_^_</span>\u65f6\u8bbe\u7f6e\u7684\u3002</p>\n",
|
||||
"<p>Check to make sure the parameter is not sharing storage with anything else </p>\n": "<p>\u68c0\u67e5\u4ee5\u786e\u4fdd\u8be5\u53c2\u6570\u4e0d\u4e0e\u5176\u4ed6\u4efb\u4f55\u5185\u5bb9\u5171\u4eab\u5b58\u50a8</p>\n",
|
||||
"<p>Clean the gradients </p>\n": "<p>\u6e05\u7406\u6e10\u53d8</p>\n",
|
||||
"<p>Cleanup the normal parameters </p>\n": "<p>\u6e05\u7406\u666e\u901a\u53c2\u6570</p>\n",
|
||||
"<p>Cleanup the parameters of the layer.</p>\n<p><em>Skip cleaning up if autograd is enabled and this is the last layer in the network, because we will need to fetch the parameters again for the backward pass.</em> </p>\n": "<p>\u6e05\u7406\u56fe\u5c42\u7684\u53c2\u6570\u3002</p>\n<p><em>\u5982\u679c\u542f\u7528\u4e86 autograd\uff0c\u5e76\u4e14\u8fd9\u662f\u7f51\u7edc\u4e2d\u7684\u6700\u540e\u4e00\u5c42\uff0c\u5219\u8df3\u8fc7\u6e05\u7406\uff0c\u56e0\u4e3a\u6211\u4eec\u9700\u8981\u518d\u6b21\u83b7\u53d6\u53c2\u6570\u624d\u80fd\u8fdb\u884c\u53cd\u5411\u4f20\u9012\u3002</em></p>\n",
|
||||
"<p>Collect all the parameters of the layer </p>\n": "<p>\u6536\u96c6\u56fe\u5c42\u7684\u6240\u6709\u53c2\u6570</p>\n",
|
||||
"<p>Collect gradients </p>\n": "<p>\u6536\u96c6\u6e10\u53d8</p>\n",
|
||||
"<p>Collect the chunk data </p>\n": "<p>\u6536\u96c6\u533a\u5757\u6570\u636e</p>\n",
|
||||
"<p>Collect the individual parameter tensors </p>\n": "<p>\u6536\u96c6\u5355\u4e2a\u53c2\u6570\u5f20\u91cf</p>\n",
|
||||
"<p>Compute the outputs of the current layer </p>\n": "<p>\u8ba1\u7b97\u5f53\u524d\u56fe\u5c42\u7684\u8f93\u51fa</p>\n",
|
||||
"<p>Concatenate all the parameters and pad it </p>\n": "<p>\u8fde\u63a5\u6240\u6709\u53c2\u6570\u5e76\u586b\u5145\u5b83</p>\n",
|
||||
"<p>Concatenate both trainable and fixed chunks </p>\n": "<p>\u8fde\u63a5\u53ef\u8bad\u7ec3\u548c\u56fa\u5b9a\u533a\u5757</p>\n",
|
||||
"<p>Concatenate both trainable and fixed params </p>\n": "<p>\u8fde\u63a5\u53ef\u8bad\u7ec3\u53c2\u6570\u548c\u56fa\u5b9a\u53c2\u6570</p>\n",
|
||||
"<p>Create an empty padding tensor </p>\n": "<p>\u521b\u5efa\u4e00\u4e2a\u7a7a\u7684\u586b\u5145\u5f20\u91cf</p>\n",
|
||||
"<p>Create an empty tensor to receive the parameters </p>\n": "<p>\u521b\u5efa\u4e00\u4e2a\u7a7a\u5f20\u91cf\u6765\u63a5\u6536\u53c2\u6570</p>\n",
|
||||
"<p>Create an empty tensor to receive the sizes </p>\n": "<p>\u521b\u5efa\u4e00\u4e2a\u7a7a\u5f20\u91cf\u6765\u63a5\u6536\u5927\u5c0f</p>\n",
|
||||
"<p>Create parameters for trainable (<span translate=no>_^_0_^_</span>) and fixed (<span translate=no>_^_1_^_</span>) parameters to be stored in current device/node </p>\n": "<p>\u4e3a\u8981\u5b58\u50a8\u5728\u5f53\u524d\u8bbe\u5907/\u8282\u70b9\u4e2d\u7684\u53ef\u8bad\u7ec3 (<span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>) \u548c fixed () \u53c2\u6570\u521b\u5efa\u53c2\u6570</p>\n",
|
||||
"<p>Data type of the layer </p>\n": "<p>\u56fe\u5c42\u7684\u6570\u636e\u7c7b\u578b</p>\n",
|
||||
"<p>Decrement the hooks counter </p>\n": "<p>\u51cf\u5c11\u94a9\u5b50\u8ba1\u6570\u5668</p>\n",
|
||||
"<p>Device of the layer </p>\n": "<p>\u8be5\u5c42\u7684\u8bbe\u5907</p>\n",
|
||||
"<p>Each shard keeps parameters in <span translate=no>_^_0_^_</span> list. The <span translate=no>_^_1_^_</span> is for trainable parameters and <span translate=no>_^_2_^_</span> is for fixed parameters. </p>\n": "<p>\u6bcf\u4e2a\u5206\u7247\u90fd\u5c06\u53c2\u6570\u4fdd\u5b58\u5728<span translate=no>_^_0_^_</span>\u5217\u8868\u4e2d\u3002\u7528\u4e8e<span translate=no>_^_1_^_</span>\u53ef\u8bad\u7ec3\u7684\u53c2\u6570\uff0c<span translate=no>_^_2_^_</span>\u7528\u4e8e\u56fa\u5b9a\u53c2\u6570\u3002</p>\n",
|
||||
"<p>Empty tensor to accumulate the gradients of the current shard </p>\n": "<p>\u7a7a\u5f20\u91cf\u7d2f\u79ef\u5f53\u524d\u5206\u7247\u7684\u68af\u5ea6</p>\n",
|
||||
"<p>Fetch all the parameters of the current node. This gets called by the previous layer so this call is just to make sure parameters are fetched. </p>\n": "<p>\u83b7\u53d6\u5f53\u524d\u8282\u70b9\u7684\u6240\u6709\u53c2\u6570\u3002\u8fd9\u88ab\u524d\u4e00\u5c42\u8c03\u7528\uff0c\u6240\u4ee5\u8fd9\u4e2a\u8c03\u7528\u53ea\u662f\u4e3a\u4e86\u786e\u4fdd\u53c2\u6570\u88ab\u6293\u53d6\u3002</p>\n",
|
||||
"<p>Forward pass </p>\n": "<p>\u5411\u524d\u4f20\u7403</p>\n",
|
||||
"<p>Gather the parameters from all the nodes/devices </p>\n": "<p>\u4ece\u6240\u6709\u8282\u70b9/\u8bbe\u5907\u6536\u96c6\u53c2\u6570</p>\n",
|
||||
"<p>Get a handle to add the backward hook. <a href=\"https://amsword.medium.com/understanding-pytorchs-autograd-with-grad-fn-and-next-functions-b2c4836daa00\">This blog discusses about <span translate=no>_^_0_^_</span></a>. </p>\n": "<p>\u83b7\u53d6\u4e00\u4e2a\u624b\u67c4\u6765\u6dfb\u52a0\u5411\u540e\u94a9\u3002<a href=\"https://amsword.medium.com/understanding-pytorchs-autograd-with-grad-fn-and-next-functions-b2c4836daa00\">\u8fd9\u7bc7\u535a\u5ba2\u8ba8\u8bba<span translate=no>_^_0_^_</span></a>\u4e86.</p>\n",
|
||||
"<p>Handle a backward event </p>\n": "<p>\u5904\u7406\u5411\u540e\u4e8b\u4ef6</p>\n",
|
||||
"<p>If all the hooks (including the module hook) have been called, then we can back up gradients and clean up the parameters. </p>\n": "<p>\u5982\u679c\u6240\u6709\u7684\u94a9\u5b50\uff08\u5305\u62ec\u6a21\u5757\u94a9\u5b50\uff09\u90fd\u88ab\u8c03\u7528\u4e86\uff0c\u90a3\u4e48\u6211\u4eec\u53ef\u4ee5\u5907\u4efd\u6e10\u53d8\u5e76\u6e05\u7406\u53c2\u6570\u3002</p>\n",
|
||||
"<p>If it is not divisible by <span translate=no>_^_0_^_</span>, pad it </p>\n": "<p>\u5982\u679c\u5b83\u4e0d\u80fd\u88ab\u6574\u9664<span translate=no>_^_0_^_</span>\uff0c\u8bf7\u586b\u5145\u5b83</p>\n",
|
||||
"<p>If there are no parameters, skip </p>\n": "<p>\u5982\u679c\u6ca1\u6709\u53c2\u6570\uff0c\u8bf7\u8df3\u8fc7</p>\n",
|
||||
"<p>Increment the number of hooks added </p>\n": "<p>\u589e\u52a0\u6dfb\u52a0\u7684\u94a9\u5b50\u6570\u91cf</p>\n",
|
||||
"<p>Initialize the properties </p>\n": "<p>\u521d\u59cb\u5316\u5c5e\u6027</p>\n",
|
||||
"<p>Iterate through all parameters </p>\n": "<p>\u904d\u5386\u6240\u6709\u53c2\u6570</p>\n",
|
||||
"<p>Iterate through model parameters and assign the values from the continuous tensor </p>\n": "<p>\u904d\u5386\u6a21\u578b\u53c2\u6570\u5e76\u5206\u914d\u6765\u81ea\u8fde\u7eed\u5f20\u91cf\u7684\u503c</p>\n",
|
||||
"<p>Iterate through trainable parameters </p>\n": "<p>\u904d\u5386\u53ef\u8bad\u7ec3\u7684\u53c2\u6570</p>\n",
|
||||
"<p>Keep a reference to the handle </p>\n": "<p>\u4fdd\u7559\u5bf9\u624b\u67c4\u7684\u5f15\u7528</p>\n",
|
||||
"<p>List of layers right after this layer </p>\n": "<p>\u7d27\u968f\u6b64\u56fe\u5c42\u4e4b\u540e\u7684\u56fe\u5c42\u5217\u8868</p>\n",
|
||||
"<p>List of layers right before this layer </p>\n": "<p>\u6b64\u56fe\u5c42\u4e4b\u524d\u7684\u56fe\u5c42\u5217\u8868</p>\n",
|
||||
"<p>Loop through trainable parameters of the current layer </p>\n": "<p>\u5faa\u73af\u6d4f\u89c8\u5f53\u524d\u56fe\u5c42\u7684\u53ef\u8bad\u7ec3\u53c2\u6570</p>\n",
|
||||
"<p>Make sure a hook hasn't already been added </p>\n": "<p>\u786e\u4fdd\u5c1a\u672a\u6dfb\u52a0\u6302\u94a9</p>\n",
|
||||
"<p>Make sure gradient back up is complete </p>\n": "<p>\u786e\u4fdd\u6e10\u53d8\u5907\u4efd\u5df2\u5b8c\u6210</p>\n",
|
||||
"<p>Make sure the parameter has no gradient data </p>\n": "<p>\u786e\u4fdd\u53c2\u6570\u6ca1\u6709\u68af\u5ea6\u6570\u636e</p>\n",
|
||||
"<p>Merge and pad trainable (<span translate=no>_^_0_^_</span>) and fixed (<span translate=no>_^_1_^_</span>) parameters </p>\n": "<p>\u5408\u5e76\u548c\u586b\u5145\u53ef\u8bad\u7ec3 (<span translate=no>_^_0_^_</span>) \u548c fixed (<span translate=no>_^_1_^_</span>) \u53c2\u6570</p>\n",
|
||||
"<p>Number of backward hooks added </p>\n": "<p>\u6dfb\u52a0\u7684\u5411\u540e\u94a9\u5b50\u6570\u91cf</p>\n",
|
||||
"<p>Number of nodes/devices the data is sharded across </p>\n": "<p>\u5206\u7247\u6570\u636e\u7684\u8282\u70b9/\u8bbe\u5907\u6570\u91cf</p>\n",
|
||||
"<p>Offset of the continuous buffer </p>\n": "<p>\u8fde\u7eed\u7f13\u51b2\u533a\u7684\u504f\u79fb\u91cf</p>\n",
|
||||
"<p>Offset of the continuous tensor </p>\n": "<p>\u8fde\u7eed\u5f20\u91cf\u7684\u504f\u79fb\u91cf</p>\n",
|
||||
"<p>Original parameter shape </p>\n": "<p>\u539f\u59cb\u53c2\u6570\u5f62\u72b6</p>\n",
|
||||
"<p>Otherwise, no need to pad </p>\n": "<p>\u5426\u5219\uff0c\u65e0\u9700\u586b\u5145</p>\n",
|
||||
"<p>Receive the parameters </p>\n": "<p>\u63a5\u6536\u53c2\u6570</p>\n",
|
||||
"<p>Receive the sizes </p>\n": "<p>\u6536\u5230\u5c3a\u7801</p>\n",
|
||||
"<p>Remove the handle from the parameter </p>\n": "<p>\u4ece\u53c2\u6570\u4e2d\u79fb\u9664\u53e5\u67c4</p>\n",
|
||||
"<p>Reshape the trainable and fixed parameters to continuous tensors </p>\n": "<p>\u5c06\u53ef\u8bad\u7ec3\u548c\u56fa\u5b9a\u53c2\u6570\u91cd\u5851\u4e3a\u8fde\u7eed\u5f20\u91cf</p>\n",
|
||||
"<p>Resize the storage to <span translate=no>_^_0_^_</span>. This will release the memory used by the parameter.</p>\n<p><strong>Setting <span translate=no>_^_1_^_</span> will not release the memory, since the autograd graph keeps a reference to it.</strong> </p>\n": "<p>\u5c06\u5b58\u50a8\u7a7a\u95f4\u8c03\u6574\u4e3a<span translate=no>_^_0_^_</span>\u3002\u8fd9\u5c06\u91ca\u653e\u53c2\u6570\u4f7f\u7528\u7684\u5185\u5b58\u3002</p>\n<p><strong>\u8bbe\u7f6e<span translate=no>_^_1_^_</span>\u4e0d\u4f1a\u91ca\u653e\u5185\u5b58\uff0c\u56e0\u4e3a autograd \u56fe\u5f62\u4f1a\u4fdd\u7559\u5bf9\u5b83\u7684\u5f15\u7528\u3002</strong></p>\n",
|
||||
"<p>Return and empty list if there are no trainable parameters </p>\n": "<p>\u5982\u679c\u6ca1\u6709\u53ef\u8bad\u7ec3\u7684\u53c2\u6570\uff0c\u5219\u8fd4\u56de\u7a7a\u5217\u8868</p>\n",
|
||||
"<p>Return the list of trainable chunks from each layer </p>\n": "<p>\u8fd4\u56de\u6bcf\u5c42\u53ef\u8bad\u7ec3\u533a\u5757\u7684\u5217\u8868</p>\n",
|
||||
"<p>Return the trainable chunk as a list </p>\n": "<p>\u5c06\u53ef\u8bad\u7ec3\u533a\u5757\u4f5c\u4e3a\u5217\u8868\u8fd4\u56de</p>\n",
|
||||
"<p>Scatter them to all the nodes/devices </p>\n": "<p>\u5c06\u5b83\u4eec\u5206\u6563\u5230\u6240\u6709\u8282\u70b9/\u8bbe\u5907</p>\n",
|
||||
"<p>Separate parameters as trainable and fixed </p>\n": "<p>\u5c06\u53c2\u6570\u5206\u4e3a\u53ef\u8bad\u7ec3\u548c\u56fa\u5b9a</p>\n",
|
||||
"<p>Set layer index </p>\n": "<p>\u8bbe\u7f6e\u56fe\u5c42\u7d22\u5f15</p>\n",
|
||||
"<p>Set preceding layers </p>\n": "<p>\u8bbe\u7f6e\u524d\u9762\u7684\u56fe\u5c42</p>\n",
|
||||
"<p>Set proceeding layers </p>\n": "<p>\u8bbe\u7f6e\u540e\u7eed\u56fe\u5c42</p>\n",
|
||||
"<p>Set streams </p>\n": "<p>\u8bbe\u7f6e\u76f4\u64ad</p>\n",
|
||||
"<p>Set the chunk gradients. This is what the optimizer sees. </p>\n": "<p>\u8bbe\u7f6e\u5206\u5757\u6e10\u53d8\u3002\u8fd9\u5c31\u662f\u4f18\u5316\u5668\u6240\u770b\u5230\u7684\u3002</p>\n",
|
||||
"<p>Set the flag </p>\n": "<p>\u8bbe\u7f6e\u65d7\u5e1c</p>\n",
|
||||
"<p>Set the flag to indicate that the parameters are not fetched </p>\n": "<p>\u8bbe\u7f6e\u6807\u5fd7\u4ee5\u6307\u793a\u672a\u8bfb\u53d6\u53c2\u6570</p>\n",
|
||||
"<p>Set the streams and preceding and proceeding layers for each <span translate=no>_^_0_^_</span> layer </p>\n": "<p>\u4e3a\u6bcf\u4e2a\u5c42\u8bbe\u7f6e\u6d41\u4ee5\u53ca\u524d\u9762\u548c\u540e\u9762\u7684<span translate=no>_^_0_^_</span>\u5c42</p>\n",
|
||||
"<p>Skip if there are no trainable parameters </p>\n": "<p>\u5982\u679c\u6ca1\u6709\u53ef\u8bad\u7ec3\u7684\u53c2\u6570\uff0c\u5219\u8df3\u8fc7</p>\n",
|
||||
"<p>Skip if there's nothing to fetch or share. </p>\n": "<p>\u5982\u679c\u6ca1\u6709\u8981\u83b7\u53d6\u6216\u5171\u4eab\u7684\u5185\u5bb9\uff0c\u8bf7\u8df3\u8fc7\u3002</p>\n",
|
||||
"<p>Skip is already fetched </p>\n": "<p>\u5df2\u83b7\u53d6 Skip</p>\n",
|
||||
"<p>Split the continuous buffer into number of nodes. These splits are views of `buffer'. </p>\n": "<p>\u5c06\u8fde\u7eed\u7f13\u51b2\u533a\u62c6\u5206\u4e3a\u591a\u4e2a\u8282\u70b9\u3002\u8fd9\u4e9b\u62c6\u5206\u662f \u201c\u7f13\u51b2\u533a\u201d \u7684\u89c6\u56fe\u3002</p>\n",
|
||||
"<p>Split the continuous buffer into the number of nodes. These splits are views of `buffer'. </p>\n": "<p>\u5c06\u8fde\u7eed\u7f13\u51b2\u533a\u62c6\u5206\u4e3a\u8282\u70b9\u6570\u3002\u8fd9\u4e9b\u62c6\u5206\u662f \u201c\u7f13\u51b2\u533a\u201d \u7684\u89c6\u56fe\u3002</p>\n",
|
||||
"<p>Split the gathered parameters into the trainable and fixed chunks </p>\n": "<p>\u5c06\u6536\u96c6\u7684\u53c2\u6570\u62c6\u5206\u4e3a\u53ef\u8bad\u7ec3\u7684\u548c\u56fa\u5b9a\u7684\u533a\u5757</p>\n",
|
||||
"<p>Start fetch parameters of the previous layer, because autograd will next process the gradients of it. </p>\n": "<p>\u5f00\u59cb\u83b7\u53d6\u524d\u4e00\u56fe\u5c42\u7684\u53c2\u6570\uff0c\u56e0\u4e3a autograd \u63a5\u4e0b\u6765\u5c06\u5904\u7406\u8be5\u56fe\u5c42\u7684\u6e10\u53d8\u3002</p>\n",
|
||||
"<p>Start fetching parameters of the proceeding layers, so that they will fetch them which the current layer does its computations. </p>\n": "<p>\u5f00\u59cb\u83b7\u53d6\u540e\u7eed\u5c42\u7684\u53c2\u6570\uff0c\u4ee5\u4fbf\u5b83\u4eec\u5c06\u83b7\u53d6\u5f53\u524d\u5c42\u8fdb\u884c\u8ba1\u7b97\u7684\u53c2\u6570\u3002</p>\n",
|
||||
"<p>Store list of modules </p>\n": "<p>\u5b58\u50a8\u6a21\u5757\u6e05\u5355</p>\n",
|
||||
"<p>Store the shape of the parameters because we need it later to reconstruct them </p>\n": "<p>\u5b58\u50a8\u53c2\u6570\u7684\u5f62\u72b6\uff0c\u56e0\u4e3a\u6211\u4eec\u7a0d\u540e\u9700\u8981\u5b83\u6765\u91cd\u5efa\u5b83\u4eec</p>\n",
|
||||
"<p>The <span translate=no>_^_0_^_</span> node will calculate the size each device/node should store, and distribute the parameters accordingly. </p>\n": "<p>\u8be5<span translate=no>_^_0_^_</span>\u8282\u70b9\u5c06\u8ba1\u7b97\u6bcf\u4e2a\u8bbe\u5907/\u8282\u70b9\u5e94\u5b58\u50a8\u7684\u5927\u5c0f\uff0c\u5e76\u76f8\u5e94\u5730\u5206\u914d\u53c2\u6570\u3002</p>\n",
|
||||
"<p>The first chunk is for trainable parameters. </p>\n": "<p>\u7b2c\u4e00\u4e2a\u533a\u5757\u7528\u4e8e\u53ef\u8bad\u7ec3\u7684\u53c2\u6570\u3002</p>\n",
|
||||
"<p>The module to be wrapped </p>\n": "<p>\u8981\u5c01\u88c5\u7684\u6a21\u5757</p>\n",
|
||||
"<p>The position of the current layer; used this for debugging logs </p>\n": "<p>\u5f53\u524d\u5c42\u7684\u4f4d\u7f6e\uff1b\u7528\u4e8e\u8c03\u8bd5\u65e5\u5fd7</p>\n",
|
||||
"<p>The previous layer will start computing gradients. We need to make sure it has finished fetching params. </p>\n": "<p>\u4e0a\u4e00\u5c42\u5c06\u5f00\u59cb\u8ba1\u7b97\u68af\u5ea6\u3002\u6211\u4eec\u9700\u8981\u786e\u4fdd\u5b83\u5df2\u7ecf\u5b8c\u6210\u4e86\u53c2\u6570\u7684\u83b7\u53d6\u3002</p>\n",
|
||||
"<p>This is the list of parameters split into lists as trainable and fixed parameters. </p>\n": "<p>\u8fd9\u662f\u5206\u4e3a\u53ef\u8bad\u7ec3\u53c2\u6570\u548c\u56fa\u5b9a\u53c2\u6570\u7684\u5217\u8868\u7684\u53c2\u6570\u5217\u8868\u3002</p>\n",
|
||||
"<p>This is the sizes of the chunks in <span translate=no>_^_0_^_</span> list. </p>\n": "<p>\u8fd9\u662f<span translate=no>_^_0_^_</span>\u5217\u8868\u4e2d\u533a\u5757\u7684\u5927\u5c0f\u3002</p>\n",
|
||||
"<p>Total number of parameters </p>\n": "<p>\u53c2\u6570\u603b\u6570</p>\n",
|
||||
"<p>Update the offset </p>\n": "<p>\u66f4\u65b0\u504f\u79fb\u91cf</p>\n",
|
||||
"<p>Use <span translate=no>_^_0_^_</span> to create an autograd step which we can intercept </p>\n": "<p><span translate=no>_^_0_^_</span>\u7528\u4e8e\u521b\u5efa\u6211\u4eec\u53ef\u4ee5\u62e6\u622a\u7684 autograd \u6b65\u9aa4</p>\n",
|
||||
"<p>Use <span translate=no>_^_0_^_</span> to fetch the parameters from all the shards </p>\n": "<p><span translate=no>_^_0_^_</span>\u4f7f\u7528\u4ece\u6240\u6709\u5206\u7247\u4e2d\u83b7\u53d6\u53c2\u6570</p>\n",
|
||||
"<p>Use the backup stream to backup the gradients </p>\n": "<p>\u4f7f\u7528\u5907\u4efd\u6d41\u5907\u4efd\u6e10\u53d8</p>\n",
|
||||
"<p>Wait for operations on the parameters to complete before any new operations </p>\n": "<p>\u5728\u8fdb\u884c\u4efb\u4f55\u65b0\u64cd\u4f5c\u4e4b\u524d\uff0c\u8bf7\u7b49\u5f85\u5bf9\u53c2\u6570\u7684\u64cd\u4f5c\u5b8c\u6210</p>\n",
|
||||
"<p>Wait for parameter fetching to complete. </p>\n": "<p>\u7b49\u5f85\u53c2\u6570\u63d0\u53d6\u5b8c\u6210\u3002</p>\n",
|
||||
"<p>Wait for the gather operation to complete and then clear the references to the buffers </p>\n": "<p>\u7b49\u5f85\u6536\u96c6\u64cd\u4f5c\u5b8c\u6210\uff0c\u7136\u540e\u6e05\u9664\u5bf9\u7f13\u51b2\u533a\u7684\u5f15\u7528</p>\n",
|
||||
"<p>Wait for the operation to complete and then clear the references to the buffers </p>\n": "<p>\u7b49\u5f85\u64cd\u4f5c\u5b8c\u6210\uff0c\u7136\u540e\u6e05\u9664\u5bf9\u7f13\u51b2\u533a\u7684\u5f15\u7528</p>\n",
|
||||
"<p>Wait for the operation to complete before other operations can be performed </p>\n": "<p>\u7b49\u5f85\u64cd\u4f5c\u5b8c\u6210\u540e\u624d\u80fd\u6267\u884c\u5176\u4ed6\u64cd\u4f5c</p>\n",
|
||||
"<p>Wait for the operations to complete before other operations can be performed </p>\n": "<p>\u7b49\u5f85\u64cd\u4f5c\u5b8c\u6210\u540e\u624d\u80fd\u6267\u884c\u5176\u4ed6\u64cd\u4f5c</p>\n",
|
||||
"<p>Whether parameters have been fetched </p>\n": "<p>\u53c2\u6570\u662f\u5426\u5df2\u83b7\u53d6</p>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> List of <span translate=no>_^_1_^_</span> layers</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u56fe\u5c42\u5217\u8868</li></ul>\n",
|
||||
"<ul><li><span translate=no>_^_0_^_</span> The module to be wrapped. </li>\n<li><span translate=no>_^_1_^_</span> The rank of the current node. </li>\n<li><span translate=no>_^_2_^_</span> The number of nodes/devices the data is sharded across. </li>\n<li><span translate=no>_^_3_^_</span> The device of the layer. </li>\n<li><span translate=no>_^_4_^_</span> The data type of the layer.</li></ul>\n": "<ul><li><span translate=no>_^_0_^_</span>\u8981\u5c01\u88c5\u7684\u6a21\u5757\u3002</li>\n<li><span translate=no>_^_1_^_</span>\u5f53\u524d\u8282\u70b9\u7684\u7b49\u7ea7\u3002</li>\n<li><span translate=no>_^_2_^_</span>\u5206\u7247\u6570\u636e\u7684\u8282\u70b9/\u8bbe\u5907\u6570\u91cf\u3002</li>\n<li><span translate=no>_^_3_^_</span>\u5c42\u7684\u8bbe\u5907\u3002</li>\n<li><span translate=no>_^_4_^_</span>\u56fe\u5c42\u7684\u6570\u636e\u7c7b\u578b\u3002</li></ul>\n",
|
||||
"This is an implementation of Zero-DP Memory Optimization written in PyTorch.": "\u8fd9\u662f\u7528 PyTorch \u7f16\u5199\u7684\u96f6 DP \u5185\u5b58\u4f18\u5316\u7684\u5b9e\u73b0\u3002",
|
||||
"Zero-DP Memory Optimization": "\u96f6 DP \u5185\u5b58\u4f18\u5316"
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"<h1>Finetune <a href=\"../../neox/index.html\">GPT-NeoX</a> with <a href=\"index.html\">Zero3 memory optimizer</a></h1>\n<p>This script trains the bias parameters of the <a href=\"../../neox/model.html\">GPT-NeoX model</a> on multiple devices with Zero-DP Memory Optimization.</p>\n": "<h1><a href=\"index.html\">Zero3 \u30e1\u30e2\u30ea\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u306b\u3088\u308b <a href=\"../../neox/index.html\">GPT-NeOx</a> \u306e\u5fae\u8abf\u6574</a></h1>\n<p>\u3053\u306e\u30b9\u30af\u30ea\u30d7\u30c8\u306f\u3001<a href=\"../../neox/model.html\">\u30bc\u30edDP\u30e1\u30e2\u30ea\u6700\u9069\u5316\u6a5f\u80fd\u3092\u5099\u3048\u305f\u8907\u6570\u306e\u30c7\u30d0\u30a4\u30b9\u4e0a\u3067GPT-Neox\u30e2\u30c7\u30eb\u306e\u30d0\u30a4\u30a2\u30b9\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h4>Create the model with Zero3 memory optimizer</h4>\n": "<h4>Zero3 \u30e1\u30e2\u30ea\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3067\u30e2\u30c7\u30eb\u3092\u4f5c\u6210</h4>\n",
|
||||
"<h4>Run the training on the node with rank <span translate=no>_^_0_^_</span>.</h4>\n": "<h4>\u30e9\u30f3\u30af\u306e\u3042\u308b\u30ce\u30fc\u30c9\u3067\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u5b9f\u884c\u3057\u307e\u3059<span translate=no>_^_0_^_</span>\u3002</h4>\n",
|
||||
"<h4>Set the optimizers for the model</h4>\n<p>Note that we pass the sharded parameters from <span translate=no>_^_0_^_</span>.</p>\n": "<h4>\u30e2\u30c7\u30eb\u306e\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u3092\u8a2d\u5b9a\u3057\u307e\u3059\u3002</h4>\n<p><span translate=no>_^_0_^_</span>\u30b7\u30e3\u30fc\u30c9\u3055\u308c\u305f\u30d1\u30e9\u30e1\u30fc\u30bf\u306f\u304b\u3089\u6e21\u3059\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create a sequential model </p>\n": "<p>\u30b7\u30fc\u30b1\u30f3\u30b7\u30e3\u30eb\u30e2\u30c7\u30eb\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u69cb\u6210\u306e\u4f5c\u6210</p>\n",
|
||||
"<p>Create the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Initialize PyTorch distributed process group </p>\n": "<p>PyTorch \u5206\u6563\u30d7\u30ed\u30bb\u30b9\u30b0\u30eb\u30fc\u30d7\u3092\u521d\u671f\u5316</p>\n",
|
||||
"<p>Initialize the model. Do this before the loop for cleaner logs. </p>\n": "<p>\u30e2\u30c7\u30eb\u3092\u521d\u671f\u5316\u3057\u307e\u3059\u3002\u3053\u308c\u3092\u30eb\u30fc\u30d7\u306e\u524d\u306b\u884c\u3063\u3066\u3001\u30ed\u30b0\u3092\u30af\u30ea\u30fc\u30f3\u30a2\u30c3\u30d7\u3057\u3066\u304f\u3060\u3055\u3044\u3002</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u69cb\u6210\u3092\u30ed\u30fc\u30c9</p>\n",
|
||||
"<p>Log the machine configurations </p>\n": "<p>\u30de\u30b7\u30f3\u69cb\u6210\u3092\u30ed\u30b0\u306b\u8a18\u9332\u3059\u308b</p>\n",
|
||||
"<p>Set current device </p>\n": "<p>\u73fe\u5728\u306e\u30c7\u30d0\u30a4\u30b9\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Start a process for each GPU. You will need a separate launcher if you are using multiple computers. </p>\n": "<p>GPU \u3054\u3068\u306b\u30d7\u30ed\u30bb\u30b9\u3092\u958b\u59cb\u3057\u307e\u3059\u3002\u8907\u6570\u306e\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u30fc\u3092\u4f7f\u7528\u3057\u3066\u3044\u308b\u5834\u5408\u306f\u3001\u5225\u306e\u30e9\u30f3\u30c1\u30e3\u30fc\u304c\u5fc5\u8981\u306b\u306a\u308a\u307e\u3059</p>\u3002\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u59cb\u3081\u308b</p>\n",
|
||||
"<p>To make sure the fine tuner sets the trainable parameters </p>\n": "<p>\u5fae\u8abf\u6574\u5668\u304c\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u53ef\u80fd\u306a\u30d1\u30e9\u30e1\u30fc\u30bf\u30fc\u3092\u8a2d\u5b9a\u3057\u3066\u3044\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3059\u308b\u306b\u306f</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>Use the <a href=\"../../neox/samples/finetune.html\">Pipeline Parallel Trainer configurations</a> and adapt it for Zero3 memory optimizer. </p>\n": "<p><a href=\"../../neox/samples/finetune.html\">\u30d1\u30a4\u30d7\u30e9\u30a4\u30f3\u30d1\u30e9\u30ec\u30eb\u30c8\u30ec\u30fc\u30ca\u30fc\u306e\u8a2d\u5b9a\u3092\u4f7f\u7528\u3057\u3066</a>\u3001Zero3 \u30e1\u30e2\u30ea\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc\u306b\u9069\u5fdc\u3055\u305b\u3066\u304f\u3060\u3055\u3044\u3002</p>\n",
|
||||
"<p>Wrap the layers with <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30ec\u30a4\u30e4\u30fc\u3092\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u30e9\u30c3\u30d4\u30f3\u30b0\u3057\u307e\u3059\u3002<span translate=no>_^_0_^_</span></p>\n",
|
||||
"Finetune GPT-NeoX with Zero3 memory optimizer": "Zero3 \u30e1\u30e2\u30ea\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u306b\u3088\u308b GPT-NeOx \u306e\u5fae\u8abf\u6574",
|
||||
"This script trains the bias parameters of the GPT-NeoX on multiple devices with Zero-DP Memory Optimization.": "\u3053\u306e\u30b9\u30af\u30ea\u30d7\u30c8\u306f\u3001\u30bc\u30edDP\u30e1\u30e2\u30ea\u6700\u9069\u5316\u3092\u4f7f\u7528\u3057\u3066\u8907\u6570\u306e\u30c7\u30d0\u30a4\u30b9\u3067GPT-Neox\u306e\u30d0\u30a4\u30a2\u30b9\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002"
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"<h1>Finetune <a href=\"../../neox/index.html\">GPT-NeoX</a> with <a href=\"index.html\">Zero3 memory optimizer</a></h1>\n<p>This script trains the bias parameters of the <a href=\"../../neox/model.html\">GPT-NeoX model</a> on multiple devices with Zero-DP Memory Optimization.</p>\n": "<h1><a href=\"index.html\">Zero3 \u0db8\u0dad\u0d9a \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba</a> \u0dc3\u0db8\u0d9f <a href=\"../../neox/index.html\">\u0da2\u0dd3\u0db4\u0dd3\u0da7\u0dd3-\u0db1\u0dd2\u0dba\u0ddd\u0d9a\u0dca\u0dc3\u0dca</a> \u0dc6\u0dd2\u0db1\u0dd9\u0da7\u0dd4\u0db1\u0dca</h1>\n<p>\u0db8\u0dd9\u0db8\u0dc3\u0dca\u0d9a\u0dca\u0dbb\u0dd2\u0db4\u0dca\u0da7\u0dca \u0d91\u0d9a <a href=\"../../neox/model.html\">ZERO-DP \u0db8\u0dad\u0d9a \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f \u0db6\u0dc4\u0dd4 \u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dc0\u0dbd \u0da2\u0dd3\u0db4\u0dd3\u0da7\u0dd3-\u0db1\u0dd2\u0dba\u0ddd\u0d9a\u0dca\u0dc3\u0dca \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dda</a> \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2. </p>\n",
|
||||
"<h4>Create the model with Zero3 memory optimizer</h4>\n": "<h4>Zero3\u0db8\u0dad\u0d9a \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1</h4>\n",
|
||||
"<h4>Run the training on the node with rank <span translate=no>_^_0_^_</span>.</h4>\n": "<h4>\u0dad\u0dbb\u0dcf\u0dad\u0dd2\u0dbb\u0db8\u0dc3\u0db8\u0d9f \u0db1\u0ddd\u0da9\u0dca \u0db8\u0dad \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span>. </h4>\n",
|
||||
"<h4>Set the optimizers for the model</h4>\n<p>Note that we pass the sharded parameters from <span translate=no>_^_0_^_</span>.</p>\n": "<h4>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1</h4>\n<p>\u0d85\u0db4\u0dd2\u0dad\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0db8\u0dad\u0dca \u0dc0\u0db1 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span>. </p>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p>Create a sequential model </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d9a\u0dca \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Create the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize PyTorch distributed process group </p>\n": "<p>PyTorch\u0db6\u0dd9\u0daf\u0dcf \u0dc4\u0dbb\u0dd2\u0db1 \u0dbd\u0daf \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0\u0dbd\u0dd2 \u0dc3\u0db8\u0dd6\u0dc4\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize the model. Do this before the loop for cleaner logs. </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1. \u0db4\u0dd2\u0dbb\u0dd2\u0dc3\u0dd2\u0daf\u0dd4 \u0dbd\u0ddc\u0d9c\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0dbd\u0dd6\u0db4\u0dba\u0da7 \u0db4\u0dd9\u0dbb \u0db8\u0dd9\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca\u0db4\u0dd6\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Log the machine configurations </p>\n": "<p>\u0dba\u0db1\u0dca\u0dad\u0dca\u0dbb\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca \u0dbd\u0ddc\u0d9c\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set current device </p>\n": "<p>\u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Start a process for each GPU. You will need a separate launcher if you are using multiple computers. </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca GPU \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0\u0dbd\u0dd2\u0dba\u0d9a\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1. \u0d94\u0db6 \u0db6\u0dc4\u0dd4 \u0db4\u0dbb\u0dd2\u0d9c\u0dab\u0d9a \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1\u0dca\u0db1\u0dda \u0db1\u0db8\u0dca \u0d94\u0db6\u0da7 \u0dc0\u0dd9\u0db1\u0db8 \u0daf\u0dd2\u0dba\u0dad\u0dca \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dca \u0d85\u0dc0\u0dc1\u0dca\u0dba \u0dc0\u0dda. </p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>To make sure the fine tuner sets the trainable parameters </p>\n": "<p>\u0dc3\u0dd2\u0dc4\u0dd2\u0db1\u0dca\u0dc3\u0dd4\u0dc3\u0dbb\u0d9a\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc3\u0d9a\u0dc3\u0dca \u0d9a\u0dbb\u0db1 \u0db6\u0dc0\u0da7 \u0dc0\u0d9c \u0db6\u0dbd\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 </p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Use the <a href=\"../../neox/samples/finetune.html\">Pipeline Parallel Trainer configurations</a> and adapt it for Zero3 memory optimizer. </p>\n": "<p><a href=\"../../neox/samples/finetune.html\">\u0db1\u0dbd \u0db8\u0dcf\u0dbb\u0dca\u0d9c \u0dc3\u0db8\u0dcf\u0db1\u0dca\u0dad\u0dbb \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4 \u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0dba\u0db1\u0dca</a> \u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb Zero3 \u0db8\u0dad\u0d9a \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0dba \u0d85\u0db1\u0dd4\u0dc0\u0dbb\u0dca\u0dad\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Wrap the layers with <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0dca\u0dae\u0dbb\u0dc3\u0db8\u0d9f \u0d86\u0dc0\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"Finetune GPT-NeoX with Zero3 memory optimizer": "Zero3 \u0db8\u0dad\u0d9a \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f \u0da2\u0dd3\u0db4\u0dd3\u0da7\u0dd3-\u0db1\u0dd2\u0dba\u0ddd\u0d9a\u0dca\u0dc3\u0dca \u0dc6\u0dd2\u0db1\u0dd9\u0da7\u0dd4\u0db1\u0dca",
|
||||
"This script trains the bias parameters of the GPT-NeoX on multiple devices with Zero-DP Memory Optimization.": "\u0db8\u0dd9\u0db8 \u0dc3\u0dca\u0d9a\u0dca\u0dbb\u0dd2\u0db4\u0dca\u0da7\u0dca \u0d91\u0d9a \u0da2\u0dd3\u0db4\u0dd3\u0da7\u0dd3-\u0db1\u0dd2\u0dba\u0ddd\u0d9a\u0dca\u0dc3\u0dca \u0dc4\u0dd2 \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dc1\u0dd4\u0db1\u0dca\u0dba-\u0da9\u0dd3\u0db4\u0dd3 \u0db8\u0dad\u0d9a \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba \u0dc3\u0db8\u0d9f \u0db6\u0dc4\u0dd4 \u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dc0\u0dbd \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2."
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"<h1>Finetune <a href=\"../../neox/index.html\">GPT-NeoX</a> with <a href=\"index.html\">Zero3 memory optimizer</a></h1>\n<p>This script trains the bias parameters of the <a href=\"../../neox/model.html\">GPT-NeoX model</a> on multiple devices with Zero-DP Memory Optimization.</p>\n": "<h1>\u914d\u5907 <a href=\"index.html\">Zero3 \u5185\u5b58\u4f18\u5316\u5668\u7684 Finetune GPT-NEO</a> <a href=\"../../neox/index.html\">X</a></h1>\n<p>\u8be5\u811a\u672c\u4f7f\u7528\u96f6 DP \u5185\u5b58\u4f18\u5316\u529f\u80fd\u5728\u591a\u4e2a\u5668\u4ef6\u4e0a\u8bad\u7ec3 <a href=\"../../neox/model.html\">GPT-NEOX \u6a21\u578b</a>\u7684\u504f\u7f6e\u53c2\u6570\u3002</p>\n",
|
||||
"<h4>Create the model with Zero3 memory optimizer</h4>\n": "<h4>\u4f7f\u7528 Zero3 \u5185\u5b58\u4f18\u5316\u5668\u521b\u5efa\u6a21\u578b</h4>\n",
|
||||
"<h4>Run the training on the node with rank <span translate=no>_^_0_^_</span>.</h4>\n": "<h4>\u5728\u5e26\u6709 rank \u7684\u8282\u70b9\u4e0a\u8fd0\u884c\u8bad\u7ec3<span translate=no>_^_0_^_</span>\u3002</h4>\n",
|
||||
"<h4>Set the optimizers for the model</h4>\n<p>Note that we pass the sharded parameters from <span translate=no>_^_0_^_</span>.</p>\n": "<h4>\u8bbe\u7f6e\u6a21\u578b\u7684\u4f18\u5316\u5668</h4>\n<p>\u8bf7\u6ce8\u610f\uff0c\u6211\u4eec\u4ece\u4f20\u9012\u5206\u7247\u53c2\u6570<span translate=no>_^_0_^_</span>\u3002</p>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p>Create a sequential model </p>\n": "<p>\u521b\u5efa\u987a\u5e8f\u6a21\u578b</p>\n",
|
||||
"<p>Create configurations </p>\n": "<p>\u521b\u5efa\u914d\u7f6e</p>\n",
|
||||
"<p>Create the experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Initialize PyTorch distributed process group </p>\n": "<p>\u521d\u59cb\u5316 PyTorch \u5206\u5e03\u5f0f\u8fdb\u7a0b\u7ec4</p>\n",
|
||||
"<p>Initialize the model. Do this before the loop for cleaner logs. </p>\n": "<p>\u521d\u59cb\u5316\u6a21\u578b\u3002\u5728\u5faa\u73af\u4e4b\u524d\u6267\u884c\u6b64\u64cd\u4f5c\u4ee5\u83b7\u5f97\u66f4\u6e05\u6670\u7684\u65e5\u5fd7\u3002</p>\n",
|
||||
"<p>Load configurations </p>\n": "<p>\u88c5\u8f7d\u914d\u7f6e</p>\n",
|
||||
"<p>Log the machine configurations </p>\n": "<p>\u8bb0\u5f55\u8ba1\u7b97\u673a\u914d\u7f6e</p>\n",
|
||||
"<p>Set current device </p>\n": "<p>\u8bbe\u7f6e\u5f53\u524d\u8bbe\u5907</p>\n",
|
||||
"<p>Start a process for each GPU. You will need a separate launcher if you are using multiple computers. </p>\n": "<p>\u4e3a\u6bcf\u4e2a GPU \u542f\u52a8\u4e00\u4e2a\u8fdb\u7a0b\u3002\u5982\u679c\u60a8\u4f7f\u7528\u591a\u53f0\u8ba1\u7b97\u673a\uff0c\u5219\u9700\u8981\u5355\u72ec\u7684\u542f\u52a8\u5668\u3002</p>\n",
|
||||
"<p>Start the experiment </p>\n": "<p>\u5f00\u59cb\u5b9e\u9a8c</p>\n",
|
||||
"<p>To make sure the fine tuner sets the trainable parameters </p>\n": "<p>\u786e\u4fdd\u7cbe\u7ec6\u8c03\u8c10\u5668\u8bbe\u7f6e\u4e86\u53ef\u8bad\u7ec3\u7684\u53c2\u6570</p>\n",
|
||||
"<p>Train the model </p>\n": "<p>\u8bad\u7ec3\u6a21\u578b</p>\n",
|
||||
"<p>Use the <a href=\"../../neox/samples/finetune.html\">Pipeline Parallel Trainer configurations</a> and adapt it for Zero3 memory optimizer. </p>\n": "<p>\u4f7f\u7528 Pi <a href=\"../../neox/samples/finetune.html\">peline Parallel Trainer \u914d\u7f6e\u5e76</a>\u5c06\u5176\u8c03\u6574\u4e3a Zero3 \u5185\u5b58\u4f18\u5316\u5668\u3002</p>\n",
|
||||
"<p>Wrap the layers with <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5c06\u56fe\u5c42\u5305\u88f9\u8d77\u6765<span translate=no>_^_0_^_</span></p>\n",
|
||||
"Finetune GPT-NeoX with Zero3 memory optimizer": "\u914d\u5907 Zero3 \u5185\u5b58\u4f18\u5316\u5668\u7684 Finetune GPT-NEOX",
|
||||
"This script trains the bias parameters of the GPT-NeoX on multiple devices with Zero-DP Memory Optimization.": "\u8be5\u811a\u672c\u4f7f\u7528\u96f6 DP \u5185\u5b58\u4f18\u5316\u529f\u80fd\u5728\u591a\u4e2a\u5668\u4ef6\u4e0a\u8bad\u7ec3 GPT-NEOX \u7684\u504f\u7f6e\u53c2\u6570\u3002"
|
||||
}
|
||||
Reference in New Issue
Block a user