This experiment trains Proximal Policy Optimization (PPO) agent Atari Breakout game on OpenAI Gym. It runs the game environments on multiple processes to sample efficiently.
\n\n": "\u8be5\u5b9e\u9a8c\u5728OpenAI Gym\u4e0a\u8bad\u7ec3\u4e86\u8fd1\u7aef\u7b56\u7565\u4f18\u5316\uff08PPO\uff09\u4ee3\u7406Atari Breakout\u6e38\u620f\u3002\u5b83\u5728\u591a\u4e2a\u8fdb\u7a0b\u4e0a\u8fd0\u884c\u6e38\u620f\u73af\u5883\u4ee5\u9ad8\u6548\u91c7\u6837\u3002
\n\n", "Stop the workers
\n": "\u963b\u6b62\u5de5\u4eba
\n", "\n": "\n", "
_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_ keeps track of the last observation from each worker, which is the input for the model to sample the next action
\n": "_^_0_^_\u8ddf\u8e2a\u6765\u81ea\u6bcf\u4e2a worker \u7684\u6700\u540e\u4e00\u4e2a\u89c2\u6d4b\u503c\uff0c\u8fd9\u662f\u6a21\u578b\u5bf9\u4e0b\u4e00\u4e2a\u64cd\u4f5c\u8fdb\u884c\u91c7\u6837\u7684\u8f93\u5165
\n", "_^_0_^_ returns sampled from _^_1_^_
\n": "_^_0_^_\u4ece\u4e2d\u62bd\u6837\u7684\u8fd4\u56de_^_1_^_
\n", "_^_0_^_, _^_1_^_ are actions sampled from _^_2_^_
\n": "_^_0_^_\uff0c_^_1_^_\u662f\u4ece\u4e2d\u91c7\u6837\u7684\u52a8\u4f5c_^_2_^_
\n", "_^_0_^_, where _^_1_^_ is advantages sampled from _^_2_^_. Refer to sampling function in Main class below for the calculation of _^_3_^_.
\n": "_^_0_^_\uff0c\u4f18_^_1_^_\u52bf\u4ece\u54ea\u91cc\u62bd\u6837_^_2_^_\u3002\u6709\u5173\u8ba1\u7b97\uff0c\u8bf7\u53c2\u9605\u4ee5\u4e0b Main \u7c7b\u4e2d\u7684\u91c7\u6837\u51fd\u6570_^_3_^_\u3002
\n", "A fully connected layer takes the flattened frame from third convolution layer, and outputs 512 features
\n": "\u5b8c\u5168\u8fde\u63a5\u7684\u56fe\u5c42\u4ece\u7b2c\u4e09\u4e2a\u5377\u79ef\u56fe\u5c42\u83b7\u53d6\u5e73\u5766\u7684\u5e27\uff0c\u5e76\u8f93\u51fa 512 \u4e2a\u8981\u7d20
\n", "A fully connected layer to get logits for _^_0_^_
\n": "\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42\uff0c\u7528\u4e8e\u83b7\u53d6\u65e5\u5fd7_^_0_^_
\n", "A fully connected layer to get value function
\n": "\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42\u6765\u83b7\u53d6\u4ef7\u503c\u51fd\u6570
\n", "Add a new line to the screen periodically
\n": "\u5b9a\u671f\u5728\u5c4f\u5e55\u4e0a\u6dfb\u52a0\u65b0\u884c
\n", "Add to tracker
\n": "\u6dfb\u52a0\u5230\u8ffd\u8e2a\u5668
\n", "Calculate Entropy Bonus
\n_^_0_^_
\n": "\u8ba1\u7b97\u71b5\u52a0\u6210
\n_^_0_^_
\n", "Calculate gradients
\n": "\u8ba1\u7b97\u68af\u5ea6
\n", "Calculate policy loss
\n": "\u8ba1\u7b97\u4fdd\u5355\u635f\u5931
\n", "Calculate value function loss
\n": "\u8ba1\u7b97\u503c\u51fd\u6570\u635f\u5931
\n", "Clip gradients
\n": "\u526a\u8f91\u6e10\u53d8
\n", "Clipping range
\n": "\u88c1\u526a\u8303\u56f4
\n", "Configurations
\n": "\u914d\u7f6e
\n", "Create the experiment
\n": "\u521b\u5efa\u5b9e\u9a8c
\n", "Entropy bonus coefficient
\n": "\u71b5\u52a0\u6210\u7cfb\u6570
\n", "GAE with _^_0_^_ and _^_1_^_
\n": "\u4f7f\u7528_^_0_^_\u548c\u7684 GAE_^_1_^_
\n", "Get value of after the final step
\n": "\u5728\u6700\u540e\u4e00\u6b65\u4e4b\u540e\u83b7\u53d6\u7684\u503c
\n", "Initialize the trainer
\n": "\u521d\u59cb\u5316\u8bad\u7ec3\u5668
\n", "It learns faster with a higher number of epochs, but becomes a little unstable; that is, the average episode reward does not monotonically increase over time. May be reducing the clipping range might solve it.
\n": "\u968f\u7740\u65f6\u4ee3\u6570\u91cf\u7684\u589e\u52a0\uff0c\u5b83\u5b66\u4e60\u5f97\u66f4\u5feb\uff0c\u4f46\u4f1a\u53d8\u5f97\u6709\u70b9\u4e0d\u7a33\u5b9a\uff1b\u4e5f\u5c31\u662f\u8bf4\uff0c\u5e73\u5747\u5267\u96c6\u5956\u52b1\u4e0d\u4f1a\u968f\u7740\u65f6\u95f4\u7684\u63a8\u79fb\u800c\u5355\u8c03\u589e\u52a0\u3002\u53ef\u80fd\u4f1a\u7f29\u5c0f\u526a\u5207\u8303\u56f4\u53ef\u80fd\u4f1a\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\u3002
\n", "Learning rate
\n": "\u5b66\u4e60\u7387
\n", "Number of mini batches
\n": "\u5fae\u578b\u6279\u6b21\u6570
\n", "Number of steps to run on each process for a single update
\n": "\u5355\u6b21\u66f4\u65b0\u7684\u6bcf\u4e2a\u8fdb\u7a0b\u8981\u8fd0\u884c\u7684\u6b65\u9aa4\u6570
\n", "Number of updates
\n": "\u66f4\u65b0\u6b21\u6570
\n", "Number of worker processes
\n": "\u5de5\u4f5c\u8fdb\u7a0b\u6570
\n", "PPO Loss
\n": "PPO \u635f\u5931
\n", "Run and monitor the experiment
\n": "\u8fd0\u884c\u5e76\u76d1\u63a7\u5b9e\u9a8c
\n", "Sampled observations are fed into the model to get _^_0_^_ and _^_1_^_; we are treating observations as state
\n": "\u91c7\u6837\u89c2\u6d4b\u503c\u88ab\u8f93\u5165\u5230\u6a21\u578b\u4e2d\u4ee5\u83b7\u53d6_^_0_^_\u548c_^_1_^_\uff1b\u6211\u4eec\u5c06\u89c2\u6d4b\u503c\u89c6\u4e3a\u72b6\u6001
\n", "Save tracked indicators.
\n": "\u4fdd\u5b58\u8ddf\u8e2a\u7684\u6307\u6807\u3002
\n", "Scale observations from _^_0_^_ to _^_1_^_
\n": "\u5c06\u89c2\u6d4b\u503c\u4ece\u7f29\u653e_^_0_^_\u5230_^_1_^_
\n", "Select device
\n": "\u9009\u62e9\u8bbe\u5907
\n", "Set learning rate
\n": "\u8bbe\u7f6e\u5b66\u4e60\u901f\u7387
\n", "Stop the workers
\n": "\u963b\u6b62\u5de5\u4eba
\n", "The first convolution layer takes a 84x84 frame and produces a 20x20 frame
\n": "\u7b2c\u4e00\u4e2a\u5377\u79ef\u5c42\u91c7\u7528 84x84 \u5e27\u5e76\u751f\u6210 20x20 \u5e27
\n", "The second convolution layer takes a 20x20 frame and produces a 9x9 frame
\n": "\u7b2c\u4e8c\u4e2a\u5377\u79ef\u5c42\u91c7\u7528 20x20 \u5e27\u5e76\u751f\u6210 9x9 \u7684\u5e27
\n", "The third convolution layer takes a 9x9 frame and produces a 7x7 frame
\n": "\u7b2c\u4e09\u4e2a\u5377\u79ef\u5c42\u91c7\u7528 9x9 \u5e27\u5e76\u751f\u6210 7x7 \u5e27
\n", "Update parameters based on gradients
\n": "\u6839\u636e\u6e10\u53d8\u66f4\u65b0\u53c2\u6570
\n", "Value Loss
\n": "\u4ef7\u503c\u635f\u5931
\n", "Value loss coefficient
\n": "\u4ef7\u503c\u635f\u5931\u7cfb\u6570
\n", "You can change this while the experiment is running. \u2699\ufe0f Learning rate.
\n": "\u4f60\u53ef\u4ee5\u5728\u5b9e\u9a8c\u8fd0\u884c\u65f6\u66f4\u6539\u6b64\u8bbe\u7f6e\u3002\u2699\ufe0f \u5b66\u4e60\u7387\u3002
\n", "Zero out the previously calculated gradients
\n": "\u5c06\u5148\u524d\u8ba1\u7b97\u7684\u68af\u5ea6\u5f52\u96f6
\n", "calculate advantages
\n": "\u8ba1\u7b97\u4f18\u52bf
\n", "collect episode info, which is available if an episode finished; this includes total reward and length of the episode - look at _^_0_^_ to see how it works.
\n": "\u6536\u96c6\u5267\u96c6\u4fe1\u606f\uff0c\u5728\u5267\u96c6\u7ed3\u675f\u540e\u53ef\u7528\uff1b\u8fd9\u5305\u62ec\u603b\u5956\u52b1\u548c\u5267\u96c6\u957f\u5ea6\u2014\u2014\u770b\u770b_^_0_^_\u5b83\u662f\u5982\u4f55\u8fd0\u4f5c\u7684\u3002
\n", "create workers
\n": "\u521b\u5efa\u5de5\u4f5c\u4eba\u5458
\n", "for each mini batch
\n": "\u6bcf\u5c0f\u6279\u6b21
\n", "for monitoring
\n": "\u7528\u4e8e\u76d1\u63a7
\n", "get mini batch
\n": "\u83b7\u5f97\u5c0f\u6279\u91cf
\n", "get results after executing the actions
\n": "\u6267\u884c\u64cd\u4f5c\u540e\u83b7\u5f97\u7ed3\u679c
\n", "initialize tensors for observations
\n": "\u521d\u59cb\u5316\u89c2\u6d4b\u503c\u7684\u5f20\u91cf
\n", "last 100 episode information
\n": "\u6700\u8fd1 100 \u96c6\u4fe1\u606f
\n", "model
\n": "\u6a21\u578b
\n", "number of epochs to train the model with sampled data
\n": "\u4f7f\u7528\u91c7\u6837\u6570\u636e\u8bad\u7ec3\u6a21\u578b\u7684\u5468\u671f\u6570
\n", "number of mini batches
\n": "\u5fae\u578b\u6279\u6b21\u6570
\n", "number of steps to run on each process for a single update
\n": "\u5355\u6b21\u66f4\u65b0\u7684\u6bcf\u4e2a\u8fdb\u7a0b\u8981\u8fd0\u884c\u7684\u6b65\u9aa4\u6570
\n", "number of updates
\n": "\u66f4\u65b0\u6b21\u6570
\n", "number of worker processes
\n": "\u5de5\u4f5c\u8fdb\u7a0b\u7684\u6570\u91cf
\n", "optimizer
\n": "\u4f18\u5316\u8005
\n", "run sampled actions on each worker
\n": "\u5bf9\u6bcf\u4e2a worker \u8fd0\u884c\u91c7\u6837\u64cd\u4f5c
\n", "sample _^_0_^_ from each worker
\n": "\u6bcf\u4f4d\u5de5\u4f5c\u4eba\u5458_^_0_^_\u7684\u6837\u672c
\n", "sample actions from _^_0_^_ for each worker; this returns arrays of size _^_1_^_
\n": "\u6bcf\u4e2a worker_^_0_^_ \u7684\u793a\u4f8b\u64cd\u4f5c\uff1b\u8fd9\u4f1a\u8fd4\u56de\u5927\u5c0f\u6570\u7ec4_^_1_^_
\n", "sample with current policy
\n": "\u5f53\u524d\u653f\u7b56\u7684\u6837\u672c
\n", "samples are currently in _^_0_^_ table, we should flatten it for training
\n": "\u6837\u672c\u76ee\u524d\u5728_^_0_^_\u8868\u4e2d\uff0c\u6211\u4eec\u5e94\u8be5\u5c06\u5176\u538b\u5e73\u4ee5\u8fdb\u884c\u8bad\u7ec3
\n", "shuffle for each epoch
\n": "\u968f\u673a\u64ad\u653e\u6bcf\u4e2a\u65f6\u4ee3
\n", "size of a mini batch
\n": "\u5c0f\u6279\u91cf\u7684\u5927\u5c0f
\n", "total number of samples for a single update
\n": "\u5355\u6b21\u66f4\u65b0\u7684\u6837\u672c\u603b\u6570
\n", "train
\n": "\u706b\u8f66
\n", "train the model
\n": "\u8bad\u7ec3\u6a21\u578b
\n", "\u2699\ufe0f Clip range.
\n": "\u2699\ufe0f \u526a\u8f91\u8303\u56f4\u3002
\n", "\u2699\ufe0f Entropy bonus coefficient. You can change this while the experiment is running.
\n": "\u2699\ufe0f \u71b5\u52a0\u6210\u7cfb\u6570\u3002\u4f60\u53ef\u4ee5\u5728\u5b9e\u9a8c\u8fd0\u884c\u65f6\u66f4\u6539\u6b64\u8bbe\u7f6e\u3002
\n", "\u2699\ufe0f Number of epochs to train the model with sampled data. You can change this while the experiment is running.
\n": "\u2699\ufe0f \u4f7f\u7528\u91c7\u6837\u6570\u636e\u8bad\u7ec3\u6a21\u578b\u7684\u65f6\u4ee3\u6570\u3002\u4f60\u53ef\u4ee5\u5728\u5b9e\u9a8c\u8fd0\u884c\u65f6\u66f4\u6539\u6b64\u8bbe\u7f6e\u3002
\n", "\u2699\ufe0f Value loss coefficient. You can change this while the experiment is running.
\n": "\u2699\ufe0f \u4ef7\u503c\u635f\u5931\u7cfb\u6570\u3002\u4f60\u53ef\u4ee5\u5728\u5b9e\u9a8c\u8fd0\u884c\u65f6\u66f4\u6539\u6b64\u8bbe\u7f6e\u3002
\n", "Annotated implementation to train a PPO agent on Atari Breakout game.": "\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0\uff0c\u7528\u4e8e\u5728 Atari Breakout \u6e38\u620f\u4e2d\u8bad\u7ec3 PPO \u7279\u5de5\u3002", "PPO Experiment with Atari Breakout": "PPO \u4f7f\u7528 Atari Breakout \u8fdb\u884c\u5b9e\u9a8c" }