This experiment trains Proximal Policy Optimization (PPO) agent Atari Breakout game on OpenAI Gym. It runs the game environments on multiple processes to sample efficiently.
\n\n": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001OpenAI Gym\u3067\u30d7\u30ed\u30ad\u30b7\u30de\u30eb\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316\uff08PPO\uff09\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u306eAtari\u30d6\u30ec\u30a4\u30af\u30a2\u30a6\u30c8\u30b2\u30fc\u30e0\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002\u30b2\u30fc\u30e0\u74b0\u5883\u3092\u8907\u6570\u306e\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3057\u3066\u52b9\u7387\u7684\u306b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u307e\u3059\u3002
\n\n", "Stop the workers
\n": "\u52b4\u50cd\u8005\u3092\u6b62\u3081\u308d
\n", "\n": "\n", "
_^_0_^_
\n": "_^_0_^_
\n", "_^_0_^_ keeps track of the last observation from each worker, which is the input for the model to sample the next action
\n": "_^_0_^_\u5404\u30ef\u30fc\u30ab\u30fc\u304b\u3089\u306e\u6700\u5f8c\u306e\u89b3\u6e2c\u5024\u3092\u8ffd\u8de1\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u30e2\u30c7\u30eb\u304c\u6b21\u306e\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u5165\u529b\u3067\u3059
\n", "_^_0_^_ returns sampled from _^_1_^_
\n": "_^_0_^_\u304b\u3089\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u305f\u30ea\u30bf\u30fc\u30f3 _^_1_^_
\n", "_^_0_^_, _^_1_^_ are actions sampled from _^_2_^_
\n": "_^_0_^__^_1_^_\u30a2\u30af\u30b7\u30e7\u30f3\u306f\u4ee5\u4e0b\u304b\u3089\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u307e\u3059 _^_2_^_
\n", "_^_0_^_, where _^_1_^_ is advantages sampled from _^_2_^_. Refer to sampling function in Main class below for the calculation of _^_3_^_.
\n": "_^_0_^_\u3001_^_1_^__^_2_^_\u5229\u70b9\u306f\u3069\u3053\u304b\u3089\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u3066\u3044\u308b\u306e\u304b\u3002\u306e\u8a08\u7b97\u306b\u3064\u3044\u3066\u306f\u3001\u4e0b\u8a18\u306e\u30e1\u30a4\u30f3\u30af\u30e9\u30b9\u306e\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u95a2\u6570\u3092\u53c2\u7167\u3057\u3066\u304f\u3060\u3055\u3044_^_3_^_\u3002
\n", "A fully connected layer takes the flattened frame from third convolution layer, and outputs 512 features
\n": "\u5b8c\u5168\u7d50\u5408\u5c64\u306f\u30013 \u756a\u76ee\u306e\u7573\u307f\u8fbc\u307f\u5c64\u304b\u3089\u5e73\u5766\u5316\u3055\u308c\u305f\u30d5\u30ec\u30fc\u30e0\u3092\u53d6\u308a\u51fa\u3057\u3001512 \u500b\u306e\u7279\u5fb4\u3092\u51fa\u529b\u3057\u307e\u3059\u3002
\n", "A fully connected layer to get logits for _^_0_^_
\n": "\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97\u3059\u308b\u305f\u3081\u306e\u5b8c\u5168\u63a5\u7d9a\u30ec\u30a4\u30e4\u30fc _^_0_^_
\n", "A fully connected layer to get value function
\n": "\u30d0\u30ea\u30e5\u30fc\u95a2\u6570\u3092\u5f97\u308b\u305f\u3081\u306e\u5b8c\u5168\u9023\u7d50\u30ec\u30a4\u30e4\u30fc
\n", "Add a new line to the screen periodically
\n": "\u753b\u9762\u306b\u5b9a\u671f\u7684\u306b\u65b0\u3057\u3044\u884c\u3092\u8ffd\u52a0\u3057\u3066\u304f\u3060\u3055\u3044
\n", "Add to tracker
\n": "\u30c8\u30e9\u30c3\u30ab\u30fc\u306b\u8ffd\u52a0
\n", "Calculate Entropy Bonus
\n_^_0_^_
\n": "\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u30dc\u30fc\u30ca\u30b9\u306e\u8a08\u7b97
\n_^_0_^_
\n", "Calculate gradients
\n": "\u52fe\u914d\u306e\u8a08\u7b97
\n", "Calculate policy loss
\n": "\u4fdd\u967a\u5951\u7d04\u640d\u5931\u306e\u8a08\u7b97
\n", "Calculate value function loss
\n": "\u5024\u95a2\u6570\u640d\u5931\u306e\u8a08\u7b97
\n", "Clip gradients
\n": "\u30af\u30ea\u30c3\u30d7\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3
\n", "Clipping range
\n": "\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u7bc4\u56f2
\n", "Configurations
\n": "\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3
\n", "Create the experiment
\n": "\u5b9f\u9a13\u3092\u4f5c\u6210
\n", "Entropy bonus coefficient
\n": "\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u30dc\u30fc\u30ca\u30b9\u4fc2\u6570
\n", "GAE with _^_0_^_ and _^_1_^_
\n": "GATE (_^_0_^_\u304a\u3088\u3073\u4ed8\u304d) _^_1_^_
\n", "Get value of after the final step
\n": "\u6700\u5f8c\u306e\u30b9\u30c6\u30c3\u30d7\u306e\u5f8c\u306b\u5024\u3092\u53d6\u5f97
\n", "Initialize the trainer
\n": "\u30c8\u30ec\u30fc\u30ca\u30fc\u3092\u521d\u671f\u5316
\n", "It learns faster with a higher number of epochs, but becomes a little unstable; that is, the average episode reward does not monotonically increase over time. May be reducing the clipping range might solve it.
\n": "\u30a8\u30dd\u30c3\u30af\u6570\u304c\u591a\u3044\u307b\u3069\u5b66\u7fd2\u306f\u901f\u304f\u306a\u308a\u307e\u3059\u304c\u3001\u5c11\u3057\u4e0d\u5b89\u5b9a\u306b\u306a\u308a\u307e\u3059\u3002\u3064\u307e\u308a\u3001\u30a8\u30d4\u30bd\u30fc\u30c9\u306e\u5e73\u5747\u5831\u916c\u306f\u6642\u9593\u306e\u7d4c\u904e\u3068\u3068\u3082\u306b\u5358\u8abf\u306b\u5897\u52a0\u3057\u307e\u305b\u3093\u3002\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u7bc4\u56f2\u3092\u72ed\u304f\u3059\u308b\u3053\u3068\u3067\u89e3\u6c7a\u3059\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\u3002
\n", "Learning rate
\n": "\u5b66\u7fd2\u7387
\n", "Number of mini batches
\n": "\u30df\u30cb\u30d0\u30c3\u30c1\u6570
\n", "Number of steps to run on each process for a single update
\n": "1 \u56de\u306e\u66f4\u65b0\u3067\u5404\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3059\u308b\u30b9\u30c6\u30c3\u30d7\u306e\u6570
\n", "Number of updates
\n": "\u66f4\u65b0\u56de\u6570
\n", "Number of worker processes
\n": "\u30ef\u30fc\u30ab\u30fc\u30d7\u30ed\u30bb\u30b9\u306e\u6570
\n", "PPO Loss
\n": "PPO \u30ed\u30b9
\n", "Run and monitor the experiment
\n": "\u5b9f\u9a13\u306e\u5b9f\u884c\u3068\u76e3\u8996
\n", "Sampled observations are fed into the model to get _^_0_^_ and _^_1_^_; we are treating observations as state
\n": "_^_0_^_\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u305f\u89b3\u6e2c\u5024\u306f\u30e2\u30c7\u30eb\u306b\u5165\u529b\u3055\u308c\u3001\u53d6\u5f97\u3055\u308c\u307e\u3059_^_1_^_\u3002\u89b3\u6e2c\u5024\u306f\u72b6\u614b\u3068\u3057\u3066\u6271\u3044\u307e\u3059
\n", "Save tracked indicators.
\n": "\u8ffd\u8de1\u6307\u6a19\u3092\u4fdd\u5b58\u3057\u307e\u3059\u3002
\n", "Scale observations from _^_0_^_ to _^_1_^_
\n": "_^_0_^_\u89b3\u6e2c\u5024\u3092\u304b\u3089\u306b\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0 _^_1_^_
\n", "Select device
\n": "\u30c7\u30d0\u30a4\u30b9\u3092\u9078\u629e
\n", "Set learning rate
\n": "\u5b66\u7fd2\u7387\u3092\u8a2d\u5b9a
\n", "Stop the workers
\n": "\u52b4\u50cd\u8005\u3092\u6b62\u3081\u308d
\n", "The first convolution layer takes a 84x84 frame and produces a 20x20 frame
\n": "\u6700\u521d\u306e\u7573\u307f\u8fbc\u307f\u5c64\u306f 84 x 84 \u30d5\u30ec\u30fc\u30e0\u3067\u300120 x 20 \u30d5\u30ec\u30fc\u30e0\u3092\u751f\u6210\u3057\u307e\u3059\u3002
\n", "The second convolution layer takes a 20x20 frame and produces a 9x9 frame
\n": "2 \u756a\u76ee\u306e\u7573\u307f\u8fbc\u307f\u5c64\u306f 20x20 \u30d5\u30ec\u30fc\u30e0\u3067\u30019x9 \u30d5\u30ec\u30fc\u30e0\u3092\u751f\u6210\u3057\u307e\u3059\u3002
\n", "The third convolution layer takes a 9x9 frame and produces a 7x7 frame
\n": "3 \u756a\u76ee\u306e\u7573\u307f\u8fbc\u307f\u5c64\u306f 9x9 \u30d5\u30ec\u30fc\u30e0\u3067 7x7 \u30d5\u30ec\u30fc\u30e0\u3092\u751f\u6210\u3057\u307e\u3059\u3002
\n", "Update parameters based on gradients
\n": "\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306b\u57fa\u3065\u3044\u3066\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u66f4\u65b0
\n", "Value Loss
\n": "\u4fa1\u5024\u640d\u5931
\n", "Value loss coefficient
\n": "\u4fa1\u5024\u640d\u5931\u4fc2\u6570
\n", "You can change this while the experiment is running. \u2699\ufe0f Learning rate.
\n": "\u30c6\u30b9\u30c8\u306e\u5b9f\u884c\u4e2d\u306b\u3053\u308c\u3092\u5909\u66f4\u3067\u304d\u307e\u3059\u3002\u2699\ufe0f \u5b66\u7fd2\u7387\u3002
\n", "Zero out the previously calculated gradients
\n": "\u4ee5\u524d\u306b\u8a08\u7b97\u3057\u305f\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30bc\u30ed\u306b\u3057\u307e\u3059
\n", "calculate advantages
\n": "\u5229\u70b9\u3092\u8a08\u7b97
\n", "collect episode info, which is available if an episode finished; this includes total reward and length of the episode - look at _^_0_^_ to see how it works.
\n": "\u30a8\u30d4\u30bd\u30fc\u30c9\u306e\u60c5\u5831\u3092\u96c6\u3081\u307e\u3057\u3087\u3046\u3002_^_0_^_\u30a8\u30d4\u30bd\u30fc\u30c9\u304c\u7d42\u4e86\u3057\u305f\u3068\u304d\u306b\u5165\u624b\u3067\u304d\u307e\u3059\u3002\u3053\u308c\u306b\u306f\u5831\u916c\u7dcf\u984d\u3084\u30a8\u30d4\u30bd\u30fc\u30c9\u306e\u9577\u3055\u304c\u542b\u307e\u308c\u307e\u3059\u3002\u4ed5\u7d44\u307f\u3092\u78ba\u8a8d\u3057\u3066\u307f\u307e\u3057\u3087\u3046\u3002
\n", "create workers
\n": "\u30ef\u30fc\u30ab\u30fc\u3092\u4f5c\u6210
\n", "for each mini batch
\n": "\u5404\u30df\u30cb\u30d0\u30c3\u30c1\u7528
\n", "for monitoring
\n": "\u76e3\u8996\u7528
\n", "get mini batch
\n": "\u30df\u30cb\u30d0\u30c3\u30c1\u3092\u5165\u624b
\n", "get results after executing the actions
\n": "\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c\u3057\u305f\u5f8c\u306b\u7d50\u679c\u3092\u53d6\u5f97
\n", "initialize tensors for observations
\n": "\u89b3\u6e2c\u7528\u306e\u30c6\u30f3\u30bd\u30eb\u3092\u521d\u671f\u5316
\n", "last 100 episode information
\n": "\u6700\u5f8c\u306e 100 \u8a71\u306e\u60c5\u5831
\n", "model
\n": "\u30e2\u30c7\u30eb
\n", "number of epochs to train the model with sampled data
\n": "\u30b5\u30f3\u30d7\u30eb\u30c7\u30fc\u30bf\u3092\u4f7f\u7528\u3057\u3066\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u30a8\u30dd\u30c3\u30af\u306e\u6570
\n", "number of mini batches
\n": "\u30df\u30cb\u30d0\u30c3\u30c1\u6570
\n", "number of steps to run on each process for a single update
\n": "1 \u56de\u306e\u66f4\u65b0\u3067\u5404\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3059\u308b\u30b9\u30c6\u30c3\u30d7\u306e\u6570
\n", "number of updates
\n": "\u66f4\u65b0\u56de\u6570
\n", "number of worker processes
\n": "\u30ef\u30fc\u30ab\u30fc\u30d7\u30ed\u30bb\u30b9\u306e\u6570
\n", "optimizer
\n": "\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc
\n", "run sampled actions on each worker
\n": "\u5404\u30ef\u30fc\u30ab\u30fc\u3067\u30b5\u30f3\u30d7\u30eb\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c
\n", "sample _^_0_^_ from each worker
\n": "_^_0_^_\u5404\u52b4\u50cd\u8005\u304b\u3089\u306e\u30b5\u30f3\u30d7\u30eb
\n", "sample actions from _^_0_^_ for each worker; this returns arrays of size _^_1_^_
\n": "_^_0_^_\u5404\u30ef\u30fc\u30ab\u30fc\u306e\u30b5\u30f3\u30d7\u30eb\u30a2\u30af\u30b7\u30e7\u30f3\u3002\u3053\u308c\u306f\u30b5\u30a4\u30ba\u306e\u914d\u5217\u3092\u8fd4\u3057\u307e\u3059 _^_1_^_
\n", "sample with current policy
\n": "\u73fe\u884c\u30dd\u30ea\u30b7\u30fc\u306e\u30b5\u30f3\u30d7\u30eb
\n", "samples are currently in _^_0_^_ table, we should flatten it for training
\n": "_^_0_^_\u30b5\u30f3\u30d7\u30eb\u306f\u73fe\u5728\u30c6\u30fc\u30d6\u30eb\u306b\u3042\u308b\u306e\u3067\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7528\u306b\u5e73\u3089\u306b\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059
\n", "shuffle for each epoch
\n": "\u5404\u30a8\u30dd\u30c3\u30af\u306e\u30b7\u30e3\u30c3\u30d5\u30eb
\n", "size of a mini batch
\n": "\u30df\u30cb\u30d0\u30c3\u30c1\u306e\u30b5\u30a4\u30ba
\n", "total number of samples for a single update
\n": "1 \u56de\u306e\u66f4\u65b0\u3067\u306e\u30b5\u30f3\u30d7\u30eb\u306e\u7dcf\u6570
\n", "train
\n": "\u5217\u8eca
\n", "train the model
\n": "\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0
\n", "\u2699\ufe0f Clip range.
\n": "\u2699\ufe0f \u30af\u30ea\u30c3\u30d7\u30ec\u30f3\u30b8\u3002
\n", "\u2699\ufe0f Entropy bonus coefficient. You can change this while the experiment is running.
\n": "\u2699\ufe0f \u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u30dc\u30fc\u30ca\u30b9\u4fc2\u6570\u3002\u3053\u308c\u306f\u5b9f\u9a13\u306e\u5b9f\u884c\u4e2d\u306b\u5909\u66f4\u3067\u304d\u307e\u3059\u3002
\n", "\u2699\ufe0f Number of epochs to train the model with sampled data. You can change this while the experiment is running.
\n": "\u2699\ufe0f \u30b5\u30f3\u30d7\u30eb\u30c7\u30fc\u30bf\u3092\u4f7f\u7528\u3057\u3066\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u30a8\u30dd\u30c3\u30af\u306e\u6570\u3002\u3053\u308c\u306f\u5b9f\u9a13\u306e\u5b9f\u884c\u4e2d\u306b\u5909\u66f4\u3067\u304d\u307e\u3059\u3002
\n", "\u2699\ufe0f Value loss coefficient. You can change this while the experiment is running.
\n": "\u2699\ufe0f \u4fa1\u5024\u640d\u5931\u4fc2\u6570\u3002\u3053\u308c\u306f\u5b9f\u9a13\u306e\u5b9f\u884c\u4e2d\u306b\u5909\u66f4\u3067\u304d\u307e\u3059\u3002
\n", "Annotated implementation to train a PPO agent on Atari Breakout game.": "Atari Breakout \u30b2\u30fc\u30e0\u3067 PPO \u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u5b9f\u88c5\u3002", "PPO Experiment with Atari Breakout": "\u30a2\u30bf\u30ea\u30fb\u30d6\u30ec\u30a4\u30af\u30a2\u30a6\u30c8\u306b\u3088\u308bPPO\u5b9f\u9a13" }