This experiment trains a Deep Q Network (DQN) to play Atari Breakout game on OpenAI Gym. It runs the game environments on multiple processes to sample efficiently.
\n\n": "\u8be5\u5b9e\u9a8c\u8bad\u7ec3 Deep Q Network (DQN) \u5728 OpenAI Gym \u4e0a\u73a9 Atari Breakout \u6e38\u620f\u3002\u5b83\u5728\u591a\u4e2a\u8fdb\u7a0b\u4e0a\u8fd0\u884c\u6e38\u620f\u73af\u5883\u4ee5\u9ad8\u6548\u91c7\u6837\u3002
\n\n", "Stop the workers
\n": "\u963b\u6b62\u5de5\u4eba
\n", "When sampling actions we use a _^_1_^_-greedy strategy, where we take a greedy action with probabiliy _^_2_^_ and take a random action with probability _^_3_^_. We refer to _^_4_^_ as _^_5_^_.
\n": "\u52a8\u4f5c\u8fdb\u884c\u62bd\u6837\u65f6\uff0c\u6211\u4eec\u4f7f\u7528_^_1_^_-greedy\u7b56\u7565\uff0c\u5176\u4e2d\u6211\u4eec\u91c7\u53d6\u6982\u7387\u7684\u8d2a\u5a6a\u52a8\u4f5c\uff0c_^_2_^_\u5e76\u968f\u673a\u91c7\u53d6\u6982\u7387\u52a8\u4f5c_^_3_^_\u3002\u6211\u4eec\u79f0\u4e4b_^_4_^_\u4e3a_^_5_^_\u3002
\n", "_^_0_^_ for prioritized replay
\n": "_^_0_^_\u7528\u4e8e\u4f18\u5148\u91cd\u64ad
\n", "_^_0_^_ for replay buffer as a function of updates
\n": "_^_0_^_\u4f5c\u4e3a\u66f4\u65b0\u51fd\u6570\u7684\u91cd\u64ad\u7f13\u51b2\u533a
\n", "_^_0_^_, exploration fraction
\n": "_^_0_^_\uff0c\u52d8\u63a2\u5206\u6570
\n", "Add a new line to the screen periodically
\n": "\u5b9a\u671f\u5728\u5c4f\u5e55\u4e0a\u6dfb\u52a0\u65b0\u884c
\n", "Add transition to replay buffer
\n": "\u5c06\u8fc7\u6e21\u6dfb\u52a0\u5230\u91cd\u64ad\u7f13\u51b2\u533a
\n", "Calculate gradients
\n": "\u8ba1\u7b97\u68af\u5ea6
\n", "Calculate priorities for replay buffer _^_0_^_
\n": "\u8ba1\u7b97\u91cd\u64ad\u7f13\u51b2\u533a\u7684\u4f18\u5148\u7ea7_^_0_^_
\n", "Clip gradients
\n": "\u526a\u8f91\u6e10\u53d8
\n", "Collect information from each worker
\n": "\u6536\u96c6\u6bcf\u4f4d\u5458\u5de5\u7684\u4fe1\u606f
\n", "Compute Temporal Difference (TD) errors, _^_0_^_, and the loss, _^_1_^_.
\n": "\u8ba1\u7b97\u65f6\u5dee (TD) \u8bef\u5dee\u548c\u635f\u5931_^_1_^_\u3002_^_0_^_
\n", "Configurations
\n": "\u914d\u7f6e
\n", "Copy to target network initially
\n": "\u6700\u521d\u590d\u5236\u5230\u76ee\u6807\u7f51\u7edc
\n", "Create the experiment
\n": "\u521b\u5efa\u5b9e\u9a8c
\n", "Get _^_0_^_
\n": "\u5f97\u5230_^_0_^_
\n", "Get Q_values for the current observation
\n": "\u83b7\u53d6\u5f53\u524d\u89c2\u6d4b\u503c\u7684 Q_Values
\n", "Get results after executing the actions
\n": "\u6267\u884c\u64cd\u4f5c\u540e\u83b7\u53d6\u7ed3\u679c
\n", "Get the Q-values of the next state for Double Q-learning. Gradients shouldn't propagate for these
\n": "\u83b7\u53d6 \u201c\u53cc Q \u5b66\u4e60\u201d \u7684\u4e0b\u4e00\u4e2a\u72b6\u6001\u7684 Q \u503c\u3002\u68af\u5ea6\u4e0d\u5e94\u8be5\u4e3a\u8fd9\u4e9b\u4f20\u64ad
\n", "Get the predicted Q-value
\n": "\u83b7\u53d6\u9884\u6d4b\u7684 Q \u503c
\n", "Initialize the trainer
\n": "\u521d\u59cb\u5316\u8bad\u7ec3\u5668
\n", "Last 100 episode information
\n": "\u6700\u8fd1 100 \u96c6\u4fe1\u606f
\n", "Learning rate.
\n": "\u5b66\u4e60\u7387\u3002
\n", "Mini batch size
\n": "\u5c0f\u6279\u91cf
\n", "Model for sampling and training
\n": "\u91c7\u6837\u548c\u8bad\u7ec3\u6a21\u578b
\n", "Number of epochs to train the model with sampled data.
\n": "\u4f7f\u7528@@\u91c7\u6837\u6570\u636e\u8bad\u7ec3\u6a21\u578b\u7684\u5468\u671f\u6570\u3002
\n", "Number of steps to run on each process for a single update
\n": "\u5355\u6b21\u66f4\u65b0\u7684\u6bcf\u4e2a\u8fdb\u7a0b\u8981\u8fd0\u884c\u7684\u6b65\u9aa4\u6570
\n", "Number of updates
\n": "\u66f4\u65b0\u6b21\u6570
\n", "Number of worker processes
\n": "\u5de5\u4f5c\u8fdb\u7a0b\u6570
\n", "Periodically update target network
\n": "\u5b9a\u671f\u66f4\u65b0\u76ee\u6807\u7f51\u7edc
\n", "Pick the action based on _^_0_^_
\n": "\u6839\u636e\u4ee5\u4e0b\u5185\u5bb9\u9009\u62e9\u64cd\u4f5c_^_0_^_
\n", "Replay buffer with _^_0_^_. Capacity of the replay buffer must be a power of 2.
\n": "\u4f7f\u7528@@\u91cd\u64ad\u7f13\u51b2\u533a_^_0_^_\u3002\u91cd\u64ad\u7f13\u51b2\u533a\u7684\u5bb9\u91cf\u5fc5\u987b\u662f 2 \u7684\u5e42\u3002
\n", "Run and monitor the experiment
\n": "\u8fd0\u884c\u5e76\u76d1\u63a7\u5b9e\u9a8c
\n", "Run sampled actions on each worker
\n": "\u5bf9\u6bcf\u4e2a\u5de5\u4f5c\u5668\u8fd0\u884c\u91c7\u6837\u64cd\u4f5c
\n", "Sample _^_0_^_
\n": "\u6837\u672c_^_0_^_
\n", "Sample actions
\n": "\u64cd\u4f5c\u793a\u4f8b
\n", "Sample from priority replay buffer
\n": "\u6765\u81ea\u4f18\u5148\u7ea7\u91cd\u64ad\u7f13\u51b2\u533a\u7684\u6837\u672c
\n", "Sample the action with highest Q-value. This is the greedy action.
\n": "\u91c7\u6837\u5177\u6709\u6700\u9ad8 Q \u503c\u7684\u52a8\u4f5c\u3002\u8fd9\u662f\u8d2a\u5a6a\u7684\u884c\u52a8\u3002
\n", "Sample with current policy
\n": "\u5f53\u524d\u653f\u7b56\u7684\u793a\u4f8b
\n", "Sampling doesn't need gradients
\n": "\u91c7\u6837\u4e0d\u9700\u8981\u6e10\u53d8
\n", "Save tracked indicators.
\n": "\u4fdd\u5b58\u8ddf\u8e2a\u7684\u6307\u6807\u3002
\n", "Scale observations from _^_0_^_ to _^_1_^_
\n": "\u5c06\u89c2\u6d4b\u503c\u4ece\u7f29\u653e_^_0_^_\u5230_^_1_^_
\n", "Select device
\n": "\u9009\u62e9\u8bbe\u5907
\n", "Set learning rate
\n": "\u8bbe\u7f6e\u5b66\u4e60\u901f\u7387
\n", "Start training after the buffer is full
\n": "\u7f13\u51b2\u533a\u6ee1\u540e\u5f00\u59cb\u8bad\u7ec3
\n", "Stop the workers
\n": "\u963b\u6b62\u5de5\u4eba
\n", "Target model updating interval
\n": "\u76ee\u6807\u6a21\u578b\u66f4\u65b0\u95f4\u9694
\n", "This doesn't need gradients
\n": "\u8fd9\u4e0d\u9700\u8981\u6e10\u53d8
\n", "Train the model
\n": "\u8bad\u7ec3\u6a21\u578b
\n", "Uniformly sample and action
\n": "\u7edf\u4e00\u91c7\u6837\u548c\u884c\u52a8
\n", "Update parameters based on gradients
\n": "\u6839\u636e\u6e10\u53d8\u66f4\u65b0\u53c2\u6570
\n", "Update replay buffer priorities
\n": "\u66f4\u65b0\u91cd\u64ad\u7f13\u51b2\u533a\u4f18\u5148\u7ea7
\n", "Whether to chose greedy action or the random action
\n": "\u9009\u62e9\u8d2a\u5a6a\u52a8\u4f5c\u8fd8\u662f\u968f\u673a\u52a8\u4f5c
\n", "Zero out the previously calculated gradients
\n": "\u5c06\u5148\u524d\u8ba1\u7b97\u7684\u68af\u5ea6\u5f52\u96f6
\n", "create workers
\n": "\u521b\u5efa\u5de5\u4f5c\u4eba\u5458
\n", "exploration as a function of updates
\n": "\u4f5c\u4e3a\u66f4\u65b0\u51fd\u6570\u7684\u63a2\u7d22
\n", "get the initial observations
\n": "\u83b7\u5f97\u521d\u6b65\u89c2\u6d4b\u503c
\n", "initialize tensors for observations
\n": "\u521d\u59cb\u5316\u89c2\u6d4b\u503c\u7684\u5f20\u91cf
\n", "learning rate
\n": "\u5b66\u4e60\u7387
\n", "loss function
\n": "\u635f\u5931\u51fd\u6570
\n", "number of training iterations
\n": "\u8bad\u7ec3\u8fed\u4ee3\u6b21\u6570
\n", "number of updates
\n": "\u66f4\u65b0\u6b21\u6570
\n", "number of workers
\n": "\u5de5\u4f5c\u4eba\u5458\u4eba\u6570
\n", "optimizer
\n": "\u4f18\u5316\u8005
\n", "reset the workers
\n": "\u91cd\u7f6e\u5de5\u4f5c\u4eba\u5458
\n", "size of mini batch for training
\n": "\u7528\u4e8e\u8bad\u7ec3\u7684\u5fae\u578b\u6279\u6b21\u7684\u5927\u5c0f
\n", "steps sampled on each update
\n": "\u6bcf\u6b21\u66f4\u65b0\u65f6\u91c7\u6837\u7684\u6b65\u9aa4
\n", "target model to get _^_0_^_
\n": "\u8981\u83b7\u53d6\u7684\u76ee\u6807\u6a21\u578b_^_0_^_
\n", "update current observation
\n": "\u66f4\u65b0\u5f53\u524d\u89c2\u6d4b\u503c
\n", "update episode information. collect episode info, which is available if an episode finished; this includes total reward and length of the episode - look at _^_0_^_ to see how it works.
\n": "\u66f4\u65b0\u5267\u96c6\u4fe1\u606f\u3002\u6536\u96c6\u5267\u96c6\u4fe1\u606f\uff0c\u5982\u679c\u5267\u96c6\u7ed3\u675f\u5219\u53ef\u7528\uff1b\u8fd9\u5305\u62ec\u603b\u5956\u52b1\u548c\u5267\u96c6\u65f6\u957f\u2014\u2014\u770b\u770b_^_0_^_\u5b83\u662f\u5982\u4f55\u8fd0\u4f5c\u7684\u3002
\n", "update target network every 250 update
\n": "\u6bcf 250 \u6b21\u66f4\u65b0\u4e00\u6b21\u76ee\u6807\u7f51\u7edc
\n", "DQN Experiment with Atari Breakout": "\u4f7f\u7528 Atari Breakout \u8fdb\u884c DQN \u5b9e", "Implementation of DQN experiment with Atari Breakout": "\u4f7f\u7528 Atari Breakout \u5b9e\u65bd DQN \u5b9e\u9a8c" }