This experiment trains a Deep Q Network (DQN) to play Atari Breakout game on OpenAI Gym. It runs the game environments on multiple processes to sample efficiently.
\n\n": "\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001\u30c7\u30a3\u30fc\u30d7Q\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08DQN\uff09\u306bOpenAI Gym\u3067\u30a2\u30bf\u30ea\u30d6\u30ec\u30a4\u30af\u30a2\u30a6\u30c8\u30b2\u30fc\u30e0\u3092\u30d7\u30ec\u30a4\u3059\u308b\u3088\u3046\u306b\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002\u30b2\u30fc\u30e0\u74b0\u5883\u3092\u8907\u6570\u306e\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3057\u3066\u52b9\u7387\u7684\u306b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u307e\u3059\u3002
\n\n", "Stop the workers
\n": "\u52b4\u50cd\u8005\u3092\u6b62\u3081\u308d
\n", "When sampling actions we use a _^_1_^_-greedy strategy, where we take a greedy action with probabiliy _^_2_^_ and take a random action with probability _^_3_^_. We refer to _^_4_^_ as _^_5_^_.
\n": "\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3059\u308b\u3068\u304d\u306f\u3001_^_1_^_-greedy \u30b9\u30c8\u30e9\u30c6\u30b8\u30fc\u3092\u4f7f\u7528\u3057\u307e\u3059\u3002\u3064\u307e\u308a\u3001_^_2_^_\u78ba\u7387\u306e\u3042\u308b\u8caa\u6b32\u306a\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c\u3057\u3001\u78ba\u7387\u306e\u3042\u308b\u30e9\u30f3\u30c0\u30e0\u306a\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c\u3057\u307e\u3059\u3002_^_3_^__^_4_^_\u3068\u547c\u3073\u307e\u3059_^_5_^_\u3002
\n", "_^_0_^_ for prioritized replay
\n": "_^_0_^_\u512a\u5148\u518d\u751f\u7528
\n", "_^_0_^_ for replay buffer as a function of updates
\n": "_^_0_^_\u66f4\u65b0\u6a5f\u80fd\u3068\u3057\u3066\u306e\u518d\u751f\u30d0\u30c3\u30d5\u30a1\u7528
\n", "_^_0_^_, exploration fraction
\n": "_^_0_^_\u3001\u63a2\u67fb\u30d5\u30e9\u30af\u30b7\u30e7\u30f3
\n", "Add a new line to the screen periodically
\n": "\u753b\u9762\u306b\u5b9a\u671f\u7684\u306b\u65b0\u3057\u3044\u884c\u3092\u8ffd\u52a0\u3057\u3066\u304f\u3060\u3055\u3044
\n", "Add transition to replay buffer
\n": "\u518d\u751f\u30d0\u30c3\u30d5\u30a1\u306b\u30c8\u30e9\u30f3\u30b8\u30b7\u30e7\u30f3\u3092\u8ffd\u52a0
\n", "Calculate gradients
\n": "\u52fe\u914d\u306e\u8a08\u7b97
\n", "Calculate priorities for replay buffer _^_0_^_
\n": "\u518d\u751f\u30d0\u30c3\u30d5\u30a1\u306e\u512a\u5148\u5ea6\u3092\u8a08\u7b97 _^_0_^_
\n", "Clip gradients
\n": "\u30af\u30ea\u30c3\u30d7\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3
\n", "Collect information from each worker
\n": "\u5404\u4f5c\u696d\u8005\u304b\u3089\u60c5\u5831\u3092\u53ce\u96c6\u3059\u308b
\n", "Compute Temporal Difference (TD) errors, _^_0_^_, and the loss, _^_1_^_.
\n": "\u6642\u5dee (TD) \u8aa4\u5dee_^_0_^_\u3001\u304a\u3088\u3073\u640d\u5931\u3092\u8a08\u7b97\u3057\u307e\u3059\u3002_^_1_^_
\n", "Configurations
\n": "\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3
\n", "Copy to target network initially
\n": "\u6700\u521d\u306b\u30bf\u30fc\u30b2\u30c3\u30c8\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306b\u30b3\u30d4\u30fc
\n", "Create the experiment
\n": "\u5b9f\u9a13\u3092\u4f5c\u6210
\n", "Get _^_0_^_
\n": "\u53d6\u5f97 _^_0_^_
\n", "Get Q_values for the current observation
\n": "\u73fe\u5728\u306e\u89b3\u6e2c\u5024\u306e Q_value \u3092\u53d6\u5f97
\n", "Get results after executing the actions
\n": "\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c\u3057\u305f\u5f8c\u306b\u7d50\u679c\u3092\u53d6\u5f97
\n", "Get the Q-values of the next state for Double Q-learning. Gradients shouldn't propagate for these
\n": "\u4e8c\u91cdQ\u5b66\u7fd2\u306e\u6b21\u306e\u72b6\u614b\u306eQ\u5024\u3092\u53d6\u5f97\u3057\u307e\u3059\u3002\u3053\u308c\u3089\u306e\u5834\u5408\u3001\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306f\u4f1d\u64ad\u3057\u306a\u3044\u306f\u305a\u3067\u3059
\n", "Get the predicted Q-value
\n": "\u4e88\u6e2c\u3055\u308c\u305f Q \u5024\u306e\u53d6\u5f97
\n", "Initialize the trainer
\n": "\u30c8\u30ec\u30fc\u30ca\u30fc\u3092\u521d\u671f\u5316
\n", "Last 100 episode information
\n": "\u6700\u65b0100\u8a71\u306e\u60c5\u5831
\n", "Learning rate.
\n": "\u5b66\u7fd2\u7387\u3002
\n", "Mini batch size
\n": "\u30df\u30cb\u30d0\u30c3\u30c1\u30b5\u30a4\u30ba
\n", "Model for sampling and training
\n": "\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3068\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7528\u306e\u30e2\u30c7\u30eb
\n", "Number of epochs to train the model with sampled data.
\n": "\u30b5\u30f3\u30d7\u30eb\u30c7\u30fc\u30bf\u3092\u4f7f\u7528\u3057\u3066\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u30a8\u30dd\u30c3\u30af\u306e\u6570\u3002
\n", "Number of steps to run on each process for a single update
\n": "1 \u56de\u306e\u66f4\u65b0\u3067\u5404\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3059\u308b\u30b9\u30c6\u30c3\u30d7\u306e\u6570
\n", "Number of updates
\n": "\u66f4\u65b0\u56de\u6570
\n", "Number of worker processes
\n": "\u30ef\u30fc\u30ab\u30fc\u30d7\u30ed\u30bb\u30b9\u306e\u6570
\n", "Periodically update target network
\n": "\u30bf\u30fc\u30b2\u30c3\u30c8\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u5b9a\u671f\u7684\u306b\u66f4\u65b0
\n", "Pick the action based on _^_0_^_
\n": "\u4ee5\u4e0b\u306b\u57fa\u3065\u3044\u3066\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u9078\u629e\u3057\u3066\u304f\u3060\u3055\u3044 _^_0_^_
\n", "Replay buffer with _^_0_^_. Capacity of the replay buffer must be a power of 2.
\n": "\u30ea\u30d7\u30ec\u30a4\u30d0\u30c3\u30d5\u30a1\u306f_^_0_^_.\u518d\u751f\u30d0\u30c3\u30d5\u30a1\u306e\u5bb9\u91cf\u306f 2 \u306e\u7d2f\u4e57\u3067\u306a\u3051\u308c\u3070\u306a\u308a\u307e\u305b\u3093
\u3002\n", "Run and monitor the experiment
\n": "\u5b9f\u9a13\u306e\u5b9f\u884c\u3068\u76e3\u8996
\n", "Run sampled actions on each worker
\n": "\u5404\u30ef\u30fc\u30ab\u30fc\u3067\u30b5\u30f3\u30d7\u30eb\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c
\n", "Sample _^_0_^_
\n": "[\u30b5\u30f3\u30d7\u30eb] _^_0_^_
\n", "Sample actions
\n": "\u30b5\u30f3\u30d7\u30eb\u30a2\u30af\u30b7\u30e7\u30f3
\n", "Sample from priority replay buffer
\n": "\u30d7\u30e9\u30a4\u30aa\u30ea\u30c6\u30a3\u30fb\u30ea\u30d7\u30ec\u30a4\u30fb\u30d0\u30c3\u30d5\u30a1\u304b\u3089\u306e\u30b5\u30f3\u30d7\u30eb
\n", "Sample the action with highest Q-value. This is the greedy action.
\n": "Q\u5024\u304c\u6700\u3082\u9ad8\u3044\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u8caa\u6b32\u306a\u884c\u52d5\u3067\u3059
\u3002\n", "Sample with current policy
\n": "\u73fe\u5728\u306e\u30dd\u30ea\u30b7\u30fc\u3092\u542b\u3080\u30b5\u30f3\u30d7\u30eb
\n", "Sampling doesn't need gradients
\n": "\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306b\u306f\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306f\u5fc5\u8981\u3042\u308a\u307e\u305b\u3093
\n", "Save tracked indicators.
\n": "\u8ffd\u8de1\u6307\u6a19\u3092\u4fdd\u5b58\u3057\u307e\u3059\u3002
\n", "Scale observations from _^_0_^_ to _^_1_^_
\n": "_^_0_^_\u89b3\u6e2c\u5024\u3092\u304b\u3089\u306b\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0 _^_1_^_
\n", "Select device
\n": "\u30c7\u30d0\u30a4\u30b9\u3092\u9078\u629e
\n", "Set learning rate
\n": "\u5b66\u7fd2\u7387\u3092\u8a2d\u5b9a
\n", "Start training after the buffer is full
\n": "\u30d0\u30c3\u30d5\u30a1\u30fc\u304c\u3044\u3063\u3071\u3044\u306b\u306a\u3063\u305f\u3089\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3092\u958b\u59cb\u3059\u308b
\n", "Stop the workers
\n": "\u52b4\u50cd\u8005\u3092\u6b62\u3081\u308d
\n", "Target model updating interval
\n": "\u5bfe\u8c61\u30e2\u30c7\u30eb\u306e\u66f4\u65b0\u9593\u9694
\n", "This doesn't need gradients
\n": "\u3053\u308c\u306b\u306f\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306f\u5fc5\u8981\u3042\u308a\u307e\u305b\u3093
\n", "Train the model
\n": "\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0
\n", "Uniformly sample and action
\n": "\u30b5\u30f3\u30d7\u30eb\u3068\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5747\u4e00\u306b
\n", "Update parameters based on gradients
\n": "\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306b\u57fa\u3065\u3044\u3066\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u66f4\u65b0
\n", "Update replay buffer priorities
\n": "\u30ea\u30d7\u30ec\u30a4\u30d0\u30c3\u30d5\u30a1\u306e\u512a\u5148\u9806\u4f4d\u3092\u66f4\u65b0
\n", "Whether to chose greedy action or the random action
\n": "\u6b32\u5f35\u308a\u30a2\u30af\u30b7\u30e7\u30f3\u3068\u30e9\u30f3\u30c0\u30e0\u30a2\u30af\u30b7\u30e7\u30f3\u306e\u3069\u3061\u3089\u3092\u9078\u3076\u304b
\n", "Zero out the previously calculated gradients
\n": "\u4ee5\u524d\u306b\u8a08\u7b97\u3057\u305f\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30bc\u30ed\u306b\u3057\u307e\u3059
\n", "create workers
\n": "\u30ef\u30fc\u30ab\u30fc\u3092\u4f5c\u6210
\n", "exploration as a function of updates
\n": "\u66f4\u65b0\u6a5f\u80fd\u3068\u3057\u3066\u306e\u63a2\u7d22
\n", "get the initial observations
\n": "\u521d\u671f\u89b3\u6e2c\u5024\u3092\u53d6\u5f97
\n", "initialize tensors for observations
\n": "\u89b3\u6e2c\u7528\u306e\u30c6\u30f3\u30bd\u30eb\u3092\u521d\u671f\u5316
\n", "learning rate
\n": "\u5b66\u7fd2\u7387
\n", "loss function
\n": "\u640d\u5931\u95a2\u6570
\n", "number of training iterations
\n": "\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u306e\u53cd\u5fa9\u56de\u6570
\n", "number of updates
\n": "\u66f4\u65b0\u56de\u6570
\n", "number of workers
\n": "\u52b4\u50cd\u8005\u306e\u6570
\n", "optimizer
\n": "\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc
\n", "reset the workers
\n": "\u30ef\u30fc\u30ab\u30fc\u3092\u30ea\u30bb\u30c3\u30c8
\n", "size of mini batch for training
\n": "\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7528\u30df\u30cb\u30d0\u30c3\u30c1\u306e\u30b5\u30a4\u30ba
\n", "steps sampled on each update
\n": "\u66f4\u65b0\u306e\u305f\u3073\u306b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u308b\u30b9\u30c6\u30c3\u30d7
\n", "target model to get _^_0_^_
\n": "\u53d6\u5f97\u3059\u308b\u5bfe\u8c61\u30e2\u30c7\u30eb _^_0_^_
\n", "update current observation
\n": "\u73fe\u5728\u306e\u89b3\u6e2c\u5024\u3092\u66f4\u65b0
\n", "update episode information. collect episode info, which is available if an episode finished; this includes total reward and length of the episode - look at _^_0_^_ to see how it works.
\n": "\u30a8\u30d4\u30bd\u30fc\u30c9\u60c5\u5831\u3092\u66f4\u65b0\u3057\u307e\u3059\u3002\u30a8\u30d4\u30bd\u30fc\u30c9\u304c\u7d42\u4e86\u3057\u305f\u5834\u5408\u306b\u5229\u7528\u3067\u304d\u308b\u30a8\u30d4\u30bd\u30fc\u30c9\u60c5\u5831\u3092\u53ce\u96c6\u3057\u307e\u3059\u3002\u3053\u308c\u306b\u306f\u3001\u5408\u8a08\u5831\u916c\u3068\u30a8\u30d4\u30bd\u30fc\u30c9\u306e\u9577\u3055\u304c\u542b\u307e\u308c\u307e\u3059\u3002\u4ed5\u7d44\u307f\u3092\u78ba\u8a8d\u3057\u3066\u307f\u3066\u304f\u3060\u3055\u3044\u3002_^_0_^_
\n", "update target network every 250 update
\n": "250 \u56de\u306e\u66f4\u65b0\u3054\u3068\u306b\u30bf\u30fc\u30b2\u30c3\u30c8\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u66f4\u65b0
\n", "DQN Experiment with Atari Breakout": "\u30a2\u30bf\u30ea\u30fb\u30d6\u30ec\u30a4\u30af\u30a2\u30a6\u30c8\u306b\u3088\u308bDQN\u5b9f\u9a13", "Implementation of DQN experiment with Atari Breakout": "\u30a2\u30bf\u30ea\u30fb\u30d6\u30ec\u30a4\u30af\u30a2\u30a6\u30c8\u306b\u3088\u308bDQN\u5b9f\u9a13\u306e\u5b9f\u65bd" }