chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"<h1>Proximal Policy Optimization - PPO</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1707.06347\">Proximal Policy Optimization - PPO</a>.</p>\n<p>PPO is a policy gradient method for reinforcement learning. Simple policy gradient methods do a single gradient update per sample (or a set of samples). Doing multiple gradient steps for a single sample causes problems because the policy deviates too much, producing a bad policy. PPO lets us do multiple gradient updates per sample by trying to keep the policy close to the policy that was used to sample data. It does so by clipping gradient flow if the updated policy is not close to the policy used to sample the data.</p>\n<p>You can find an experiment that uses it <a href=\"experiment.html\">here</a>. The experiment uses <a href=\"gae.html\">Generalized Advantage Estimation</a>.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u8fd1\u63a5\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316-PPO</h1>\n<p><a href=\"https://arxiv.org/abs/1707.06347\">\u3053\u308c\u306f\u8fd1\u63a5\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316</a>\uff08PPO\uff09<a href=\"https://pytorch.org\">\u306ePyTorch\u5b9f\u88c5\u3067\u3059</a>\u3002</p>\n<p>PPO\u306f\u5f37\u5316\u5b66\u7fd2\u306e\u30dd\u30ea\u30b7\u30fc\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u6cd5\u3067\u3059\u3002\u30b7\u30f3\u30d7\u30eb\u306a\u30dd\u30ea\u30b7\u30fc\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30e1\u30bd\u30c3\u30c9\u3067\u306f\u3001\u30b5\u30f3\u30d7\u30eb (\u307e\u305f\u306f\u30b5\u30f3\u30d7\u30eb\u30bb\u30c3\u30c8) \u3054\u3068\u306b 1 \u56de\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u66f4\u65b0\u3092\u884c\u3044\u307e\u3059\u30021\u3064\u306e\u30b5\u30f3\u30d7\u30eb\u306b\u5bfe\u3057\u3066\u8907\u6570\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30b9\u30c6\u30c3\u30d7\u3092\u5b9f\u884c\u3059\u308b\u3068\u3001\u30dd\u30ea\u30b7\u30fc\u306e\u504f\u5dee\u304c\u5927\u304d\u3059\u304e\u3066\u4e0d\u9069\u5207\u306a\u30dd\u30ea\u30b7\u30fc\u306b\u306a\u308b\u305f\u3081\u3001\u554f\u984c\u304c\u767a\u751f\u3057\u307e\u3059\u3002PPO \u3067\u306f\u3001\u30dd\u30ea\u30b7\u30fc\u3092\u30c7\u30fc\u30bf\u306e\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306b\u4f7f\u7528\u3057\u305f\u30dd\u30ea\u30b7\u30fc\u306b\u8fd1\u3044\u72b6\u614b\u306b\u4fdd\u3064\u3053\u3068\u3067\u3001\u30b5\u30f3\u30d7\u30eb\u3054\u3068\u306b\u8907\u6570\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u66f4\u65b0\u3092\u884c\u3046\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002\u66f4\u65b0\u3055\u308c\u305f\u30dd\u30ea\u30b7\u30fc\u304c\u30c7\u30fc\u30bf\u306e\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306b\u4f7f\u7528\u3055\u308c\u305f\u30dd\u30ea\u30b7\u30fc\u306b\u5408\u308f\u306a\u3044\u5834\u5408\u306f\u3001\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30d5\u30ed\u30fc\u3092\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u3057\u3066\u66f4\u65b0\u3057\u307e\u3059</p>\u3002\n<p><a href=\"experiment.html\">\u3053\u308c\u3092\u4f7f\u3063\u305f\u5b9f\u9a13\u306f\u3053\u3061\u3089\u304b\u3089\u3054\u89a7\u3044\u305f\u3060\u3051\u307e\u3059</a>\u3002\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001<a href=\"gae.html\">\u4e00\u822c\u5316\u30a2\u30c9\u30d0\u30f3\u30c6\u30fc\u30b8\u63a8\u5b9a\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\u3002\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Clipped Value Function Loss</h2>\n<p>Similarly we clip the value function update also.</p>\n<span translate=no>_^_0_^_</span><p>Clipping makes sure the value function <span translate=no>_^_1_^_</span> doesn't deviate significantly from <span translate=no>_^_2_^_</span>.</p>\n": "<h2>\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u30d0\u30ea\u30e5\u30fc\u95a2\u6570\u306e\u640d\u5931</h2>\n<p>\u540c\u69d8\u306b\u3001\u5024\u95a2\u6570\u306e\u66f4\u65b0\u3082\u30af\u30ea\u30c3\u30d7\u3057\u307e\u3059\u3002</p>\n<span translate=no>_^_0_^_</span><p>\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u306b\u3088\u308a\u3001<span translate=no>_^_1_^_</span>\u5024\u95a2\u6570\u304c\u5927\u304d\u304f\u305a\u308c\u306a\u3044\u3088\u3046\u306b\u3057\u307e\u3059\u3002<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<h2>PPO Loss</h2>\n<p>Here's how the PPO update rule is derived.</p>\n<p>We want to maximize policy reward <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the reward, <span translate=no>_^_2_^_</span> is the policy, <span translate=no>_^_3_^_</span> is a trajectory sampled from policy, and <span translate=no>_^_4_^_</span> is the discount factor between <span translate=no>_^_5_^_</span>.</p>\n<span translate=no>_^_6_^_</span><p>So, <span translate=no>_^_7_^_</span></p>\n<p>Define discounted-future state distribution, <span translate=no>_^_8_^_</span></p>\n<p>Then,</p>\n<span translate=no>_^_9_^_</span><p>Importance sampling <span translate=no>_^_10_^_</span> from <span translate=no>_^_11_^_</span>,</p>\n<span translate=no>_^_12_^_</span><p>Then we assume <span translate=no>_^_13_^_</span> and <span translate=no>_^_14_^_</span> are similar. The error we introduce to <span translate=no>_^_15_^_</span> by this assumption is bound by the KL divergence between <span translate=no>_^_16_^_</span> and <span translate=no>_^_17_^_</span>. <a href=\"https://arxiv.org/abs/1705.10528\">Constrained Policy Optimization</a> shows the proof of this. I haven't read it.</p>\n<span translate=no>_^_18_^_</span>": "<h2>PPO \u30ed\u30b9</h2>\n<p>PPO \u66f4\u65b0\u30eb\u30fc\u30eb\u306f\u6b21\u306e\u65b9\u6cd5\u3067\u5c0e\u304d\u51fa\u3055\u308c\u307e\u3059\u3002</p>\n<p><span translate=no>_^_0_^_</span>\u3053\u3053\u3067\u3001<span translate=no>_^_1_^_</span>\u304c\u5831\u916c\u3001\u304c\u30dd\u30ea\u30b7\u30fc\u3001<span translate=no>_^_2_^_</span>\u304c\u30dd\u30ea\u30b7\u30fc\u304b\u3089\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u305f\u8ecc\u8de1\u3001<span translate=no>_^_3_^_</span><span translate=no>_^_4_^_</span>\u305d\u3057\u3066\u305d\u306e\u9593\u306e\u5272\u5f15\u4fc2\u6570\u3067\u3001\u30dd\u30ea\u30b7\u30fc\u306e\u5831\u916c\u3092\u6700\u5927\u5316\u3057\u305f\u3044\u3068\u8003\u3048\u3066\u3044\u307e\u3059\u3002<span translate=no>_^_5_^_</span></p>\n<span translate=no>_^_6_^_</span><p>\u3060\u304b\u3089\u3001<span translate=no>_^_7_^_</span></p>\n<p>\u5272\u5f15\u5f8c\u306e\u5c06\u6765\u306e\u72b6\u614b\u5206\u5e03\u3092\u5b9a\u7fa9\u3057\u3001<span translate=no>_^_8_^_</span></p>\n<p>\u6b21\u306b\u3001</p>\n<span translate=no>_^_9_^_</span><p><span translate=no>_^_10_^_</span><span translate=no>_^_11_^_</span>\u304b\u3089\u306e\u91cd\u8981\u5ea6\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0</p>\n<span translate=no>_^_12_^_</span><p>\u305d\u3046\u3059\u308b\u3068\u3001<span translate=no>_^_13_^_</span><span translate=no>_^_14_^_</span>\u4f3c\u305f\u3088\u3046\u306a\u3082\u306e\u3060\u3068\u4eee\u5b9a\u3057\u307e\u3059\u3002<span translate=no>_^_15_^_</span>\u3053\u306e\u4eee\u5b9a\u306b\u3088\u3063\u3066\u751f\u3058\u308b\u8aa4\u5dee\u306f\u3001<span translate=no>_^_16_^_</span>\u3068\u306e\u9593\u306e KL \u306e\u76f8\u9055\u306b\u3088\u3063\u3066\u6c7a\u307e\u308a\u307e\u3059\u3002<span translate=no>_^_17_^_</span><a href=\"https://arxiv.org/abs/1705.10528\">\u5236\u7d04\u4ed8\u304d\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316\u306f\u305d\u306e\u8a3c\u62e0\u3067\u3059</a>\u3002\u307e\u3060\u8aad\u3093\u3067\u306a\u3044\u3088\u3002</p>\n<span translate=no>_^_18_^_</span>",
|
||||
"<h3>Cliping the policy ratio</h3>\n<span translate=no>_^_0_^_</span><p>The ratio is clipped to be close to 1. We take the minimum so that the gradient will only pull <span translate=no>_^_1_^_</span> towards <span translate=no>_^_2_^_</span> if the ratio is not between <span translate=no>_^_3_^_</span> and <span translate=no>_^_4_^_</span>. This keeps the KL divergence between <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> constrained. Large deviation can cause performance collapse; where the policy performance drops and doesn't recover because we are sampling from a bad policy.</p>\n<p>Using the normalized advantage <span translate=no>_^_7_^_</span> introduces a bias to the policy gradient estimator, but it reduces variance a lot. </p>\n": "<h3>\u30dd\u30ea\u30b7\u30fc\u6bd4\u7387\u306e\u30af\u30ea\u30c3\u30d4\u30f3\u30b0</h3>\n<span translate=no>_^_0_^_</span><p>\u6bd4\u7387\u306f 1 \u306b\u8fd1\u3065\u304f\u3088\u3046\u306b\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u3055\u308c\u307e\u3059\u3002<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span><span translate=no>_^_3_^_</span>\u6bd4\u7387\u304c\u3068\u306e\u9593\u3067\u306a\u3044\u5834\u5408\u306b\u306e\u307f\u52fe\u914d\u304c\u50be\u304f\u3088\u3046\u306b\u6700\u5c0f\u5316\u3057\u3066\u3044\u307e\u3059<span translate=no>_^_4_^_</span>\u3002\u3053\u308c\u306b\u3088\u308a\u3001\u3068\u306e\u9593\u306e KL <span translate=no>_^_5_^_</span> \u306e\u76f8\u9055\u304c\u6291\u3048\u3089\u308c\u307e\u3059<span translate=no>_^_6_^_</span>\u3002\u5927\u304d\u306a\u504f\u5dee\u304c\u3042\u308b\u3068\u3001\u30dd\u30ea\u30b7\u30fc\u306e\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u304c\u4f4e\u4e0b\u3057\u3001\u4e0d\u9069\u5207\u306a\u30dd\u30ea\u30b7\u30fc\u304b\u3089\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u3066\u3044\u308b\u305f\u3081\u306b\u30dd\u30ea\u30b7\u30fc\u306e\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u304c\u4f4e\u4e0b\u3057\u3001\u56de\u5fa9\u3057\u306a\u3044\u5834\u5408\u304c\u3042\u308a\u307e\u3059\u3002</p>\n<p>\u6b63\u898f\u5316\u3055\u308c\u305f\u30a2\u30c9\u30d0\u30f3\u30c6\u30fc\u30b8\u3092\u4f7f\u7528\u3059\u308b\u3068\u3001<span translate=no>_^_7_^_</span>\u30dd\u30ea\u30b7\u30fc\u52fe\u914d\u63a8\u5b9a\u91cf\u306b\u504f\u308a\u304c\u751f\u3058\u307e\u3059\u304c\u3001\u5206\u6563\u306f\u5927\u5e45\u306b\u6e1b\u5c11\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>ratio <span translate=no>_^_0_^_</span>; <em>this is different from rewards</em> <span translate=no>_^_1_^_</span>. </p>\n": "<p>\u6bd4\u7387<span translate=no>_^_0_^_</span>\u3002<em>\u3053\u308c\u306f\u5831\u916c\u3068\u306f\u7570\u306a\u308a\u307e\u3059</em><span translate=no>_^_1_^_</span>\u3002</p>\n",
|
||||
"An annotated implementation of Proximal Policy Optimization - PPO algorithm in PyTorch.": "PyTorch\u306e\u8fd1\u63a5\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316-PPO\u30a2\u30eb\u30b4\u30ea\u30ba\u30e0\u306e\u6ce8\u91c8\u4ed8\u304d\u5b9f\u88c5\u3002",
|
||||
"Proximal Policy Optimization - PPO": "\u8fd1\u63a5\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316-PPO"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"<h1>Proximal Policy Optimization - PPO</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1707.06347\">Proximal Policy Optimization - PPO</a>.</p>\n<p>PPO is a policy gradient method for reinforcement learning. Simple policy gradient methods do a single gradient update per sample (or a set of samples). Doing multiple gradient steps for a single sample causes problems because the policy deviates too much, producing a bad policy. PPO lets us do multiple gradient updates per sample by trying to keep the policy close to the policy that was used to sample data. It does so by clipping gradient flow if the updated policy is not close to the policy used to sample the data.</p>\n<p>You can find an experiment that uses it <a href=\"experiment.html\">here</a>. The experiment uses <a href=\"gae.html\">Generalized Advantage Estimation</a>.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u8fd1\u7aef\u7b56\u7565\u4f18\u5316-PPO</h1>\n<p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u5b9e\u73b0\u7684<a href=\"https://arxiv.org/abs/1707.06347\">\u8fd1\u7aef\u7b56\u7565\u4f18\u5316-PPO</a>\u3002</p>\n<p>PPO \u662f\u4e00\u79cd\u7528\u4e8e\u5f3a\u5316\u5b66\u4e60\u7684\u7b56\u7565\u68af\u5ea6\u65b9\u6cd5\u3002\u7b80\u5355\u7684\u7b56\u7565\u68af\u5ea6\u65b9\u6cd5\u5bf9\u6bcf\u4e2a\u6837\u672c\uff08\u6216\u4e00\u7ec4\u6837\u672c\uff09\u8fdb\u884c\u4e00\u6b21\u68af\u5ea6\u66f4\u65b0\u3002\u5bf9\u5355\u4e2a\u6837\u672c\u6267\u884c\u591a\u4e2a\u68af\u5ea6\u6b65\u9aa4\u4f1a\u5bfc\u81f4\u95ee\u9898\uff0c\u56e0\u4e3a\u7b56\u7565\u504f\u5dee\u592a\u5927\uff0c\u4ece\u800c\u4ea7\u751f\u9519\u8bef\u7684\u7b56\u7565\u3002PPO \u5141\u8bb8\u6211\u4eec\u5728\u6bcf\u4e2a\u6837\u672c\u4e2d\u8fdb\u884c\u591a\u6b21\u68af\u5ea6\u66f4\u65b0\uff0c\u65b9\u6cd5\u662f\u5c3d\u91cf\u4f7f\u7b56\u7565\u4e0e\u7528\u4e8e\u91c7\u6837\u6570\u636e\u7684\u7b56\u7565\u4fdd\u6301\u4e00\u81f4\u3002\u5982\u679c\u66f4\u65b0\u540e\u7684\u7b56\u7565\u4e0e\u7528\u4e8e\u91c7\u6837\u6570\u636e\u7684\u7b56\u7565\u4e0d\u63a5\u8fd1\uff0c\u5219\u901a\u8fc7\u524a\u51cf\u68af\u5ea6\u6d41\u6765\u5b9e\u73b0\u6b64\u76ee\u7684\u3002</p>\n<p>\u4f60\u53ef\u4ee5<a href=\"experiment.html\">\u5728\u8fd9\u91cc</a>\u627e\u5230\u4e00\u4e2a\u4f7f\u7528\u5b83\u7684\u5b9e\u9a8c\u3002\u8be5\u5b9e\u9a8c\u4f7f\u7528<a href=\"gae.html\">\u5e7f\u4e49\u4f18\u52bf\u4f30\u8ba1</a>\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Clipped Value Function Loss</h2>\n<p>Similarly we clip the value function update also.</p>\n<span translate=no>_^_0_^_</span><p>Clipping makes sure the value function <span translate=no>_^_1_^_</span> doesn't deviate significantly from <span translate=no>_^_2_^_</span>.</p>\n": "<h2>\u524a\u51cf\u503c\u51fd\u6570\u635f\u5931</h2>\n<p>\u540c\u6837\uff0c\u6211\u4eec\u4e5f\u88c1\u526a\u503c\u51fd\u6570\u7684\u66f4\u65b0\u3002</p>\n<span translate=no>_^_0_^_</span><p>\u88c1\u526a\u53ef\u786e\u4fdd\u503c\u51fd\u6570<span translate=no>_^_1_^_</span>\u4e0d\u4f1a\u660e\u663e\u504f\u79bb<span translate=no>_^_2_^_</span>\u3002</p>\n",
|
||||
"<h2>PPO Loss</h2>\n<p>Here's how the PPO update rule is derived.</p>\n<p>We want to maximize policy reward <span translate=no>_^_0_^_</span> where <span translate=no>_^_1_^_</span> is the reward, <span translate=no>_^_2_^_</span> is the policy, <span translate=no>_^_3_^_</span> is a trajectory sampled from policy, and <span translate=no>_^_4_^_</span> is the discount factor between <span translate=no>_^_5_^_</span>.</p>\n<span translate=no>_^_6_^_</span><p>So, <span translate=no>_^_7_^_</span></p>\n<p>Define discounted-future state distribution, <span translate=no>_^_8_^_</span></p>\n<p>Then,</p>\n<span translate=no>_^_9_^_</span><p>Importance sampling <span translate=no>_^_10_^_</span> from <span translate=no>_^_11_^_</span>,</p>\n<span translate=no>_^_12_^_</span><p>Then we assume <span translate=no>_^_13_^_</span> and <span translate=no>_^_14_^_</span> are similar. The error we introduce to <span translate=no>_^_15_^_</span> by this assumption is bound by the KL divergence between <span translate=no>_^_16_^_</span> and <span translate=no>_^_17_^_</span>. <a href=\"https://arxiv.org/abs/1705.10528\">Constrained Policy Optimization</a> shows the proof of this. I haven't read it.</p>\n<span translate=no>_^_18_^_</span>": "<h2>PPO \u635f\u5931</h2>\n<p>\u4ee5\u4e0b\u662f PPO \u66f4\u65b0\u89c4\u5219\u7684\u6d3e\u751f\u65b9\u5f0f\u3002</p>\n<p>\u6211\u4eec\u5e0c\u671b\u6700\u5927\u9650\u5ea6\u5730\u63d0\u9ad8\u4fdd\u5355\u5956\u52b1<span translate=no>_^_0_^_</span>\u5728\u54ea\u91cc<span translate=no>_^_1_^_</span>\uff0c<span translate=no>_^_2_^_</span>\u5956\u52b1\u5728\u54ea\u91cc\uff0c<span translate=no>_^_3_^_</span>\u662f\u4fdd\u5355\uff0c\u662f\u4ece\u4fdd\u5355\u4e2d\u62bd\u6837\u7684\u8f68\u8ff9\uff0c<span translate=no>_^_4_^_</span>\u662f\u4ecb\u4e8e\u4e24\u8005\u4e4b\u95f4\u7684\u6298\u6263\u7cfb\u6570<span translate=no>_^_5_^_</span>\u3002</p>\n<span translate=no>_^_6_^_</span><p>\u6240\u4ee5\uff0c<span translate=no>_^_7_^_</span></p>\n<p>\u5b9a\u4e49\u6298\u6263\u672a\u6765\u72b6\u6001\u5206\u914d\uff0c<span translate=no>_^_8_^_</span></p>\n<p>\u90a3\u4e48\uff0c</p>\n<span translate=no>_^_9_^_</span><p>\u91cd\u8981\u6027\u62bd\u6837<span translate=no>_^_10_^_</span>\u6765\u81ea<span translate=no>_^_11_^_</span></p>\n<span translate=no>_^_12_^_</span><p>\u7136\u540e\u6211\u4eec\u5047\u8bbe<span translate=no>_^_13_^_</span>\u548c<span translate=no>_^_14_^_</span>\u662f\u76f8\u4f3c\u7684\u3002\u6211\u4eec<span translate=no>_^_15_^_</span>\u901a\u8fc7\u8fd9\u4e2a\u5047\u8bbe\u5f15\u5165\u7684\u8bef\u5dee\u53d7<span translate=no>_^_16_^_</span>\u548c\u4e4b\u95f4\u7684 KL \u5dee\u5f02\u7684\u7ea6\u675f<span translate=no>_^_17_^_</span>\u3002<a href=\"https://arxiv.org/abs/1705.10528\">\u7ea6\u675f\u7b56\u7565\u4f18\u5316</a>\u8bc1\u660e\u4e86\u8fd9\u4e00\u70b9\u3002\u6211\u8fd8\u6ca1\u770b\u8fc7\u3002</p>\n<span translate=no>_^_18_^_</span>",
|
||||
"<h3>Cliping the policy ratio</h3>\n<span translate=no>_^_0_^_</span><p>The ratio is clipped to be close to 1. We take the minimum so that the gradient will only pull <span translate=no>_^_1_^_</span> towards <span translate=no>_^_2_^_</span> if the ratio is not between <span translate=no>_^_3_^_</span> and <span translate=no>_^_4_^_</span>. This keeps the KL divergence between <span translate=no>_^_5_^_</span> and <span translate=no>_^_6_^_</span> constrained. Large deviation can cause performance collapse; where the policy performance drops and doesn't recover because we are sampling from a bad policy.</p>\n<p>Using the normalized advantage <span translate=no>_^_7_^_</span> introduces a bias to the policy gradient estimator, but it reduces variance a lot. </p>\n": "<h3>\u524a\u51cf\u4fdd\u5355\u6bd4\u7387</h3>\n<span translate=no>_^_0_^_</span><p>\u8be5\u6bd4\u7387\u88ab\u88c1\u526a\u4e3a\u63a5\u8fd1 1\u3002\u6211\u4eec\u53d6\u6700\u5c0f\u503c\uff0c\u4ee5\u4fbf\u53ea\u6709\u5f53\u6bd4\u7387\u4e0d\u5728<span translate=no>_^_3_^_</span>\u548c\u4e4b\u95f4\u65f6\uff0c\u68af\u5ea6\u624d\u4f1a\u62c9<span translate=no>_^_1_^_</span>\u5411<span translate=no>_^_2_^_</span><span translate=no>_^_4_^_</span>\u3002\u8fd9\u4fdd\u6301\u4e86 KL \u4e4b\u95f4\u7684\u5dee\u5f02<span translate=no>_^_5_^_</span>\u548c<span translate=no>_^_6_^_</span>\u9650\u5236\u3002\u8f83\u5927\u7684\u504f\u5dee\u53ef\u80fd\u5bfc\u81f4\u6027\u80fd\u4e0b\u964d\uff1b\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0c\u7b56\u7565\u6027\u80fd\u4f1a\u4e0b\u964d\u4e14\u65e0\u6cd5\u6062\u590d\uff0c\u56e0\u4e3a\u6211\u4eec\u6b63\u5728\u4ece\u4e0d\u826f\u7b56\u7565\u4e2d\u62bd\u6837\u3002</p>\n<p>\u4f7f\u7528\u5f52\u4e00\u5316\u4f18\u52bf\u4f1a\u7ed9\u653f\u7b56\u68af\u5ea6\u4f30\u8ba1\u5668<span translate=no>_^_7_^_</span>\u5e26\u6765\u504f\u5dee\uff0c\u4f46\u5b83\u5927\u5927\u51cf\u5c11\u4e86\u65b9\u5dee\u3002</p>\n",
|
||||
"<p>ratio <span translate=no>_^_0_^_</span>; <em>this is different from rewards</em> <span translate=no>_^_1_^_</span>. </p>\n": "<p>\u6bd4\u4f8b<span translate=no>_^_0_^_</span>\uff1b<em>\u8fd9\u4e0e\u5956\u52b1\u4e0d\u540c</em><span translate=no>_^_1_^_</span>\u3002</p>\n",
|
||||
"An annotated implementation of Proximal Policy Optimization - PPO algorithm in PyTorch.": "PyTorch \u4e2d\u8fd1\u7aef\u7b56\u7565\u4f18\u5316-PPO \u7b97\u6cd5\u7684\u5e26\u6ce8\u91ca\u5b9e\u73b0\u3002",
|
||||
"Proximal Policy Optimization - PPO": "\u8fd1\u7aef\u7b56\u7565\u4f18\u5316-PPO"
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"<h1>PPO Experiment with Atari Breakout</h1>\n<p>This experiment trains Proximal Policy Optimization (PPO) agent Atari Breakout game on OpenAI Gym. It runs the <a href=\"../game.html\">game environments on multiple processes</a> to sample efficiently.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>\u30a2\u30bf\u30ea\u30fb\u30d6\u30ec\u30a4\u30af\u30a2\u30a6\u30c8\u306b\u3088\u308bPPO\u5b9f\u9a13</h1>\n<p>\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001OpenAI Gym\u3067\u30d7\u30ed\u30ad\u30b7\u30de\u30eb\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316\uff08PPO\uff09\u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u306eAtari\u30d6\u30ec\u30a4\u30af\u30a2\u30a6\u30c8\u30b2\u30fc\u30e0\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3057\u307e\u3059\u3002<a href=\"../game.html\">\u30b2\u30fc\u30e0\u74b0\u5883\u3092\u8907\u6570\u306e\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3057\u3066\u52b9\u7387\u7684\u306b\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3057\u307e\u3059</a>\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Model</h2>\n": "<h2>\u30e2\u30c7\u30eb</h2>\n",
|
||||
"<h2>Run it</h2>\n": "<h2>\u5b9f\u884c\u3057\u3066\u304f\u3060\u3055\u3044</h2>\n",
|
||||
"<h2>Trainer</h2>\n": "<h2>\u30c8\u30ec\u30fc\u30ca\u30fc</h2>\n",
|
||||
"<h3>Calculate total loss</h3>\n": "<h3>\u7dcf\u640d\u5931\u306e\u8a08\u7b97</h3>\n",
|
||||
"<h3>Destroy</h3>\n<p>Stop the workers</p>\n": "<h3>\u7834\u58ca</h3>\n<p>\u52b4\u50cd\u8005\u3092\u6b62\u3081\u308d</p>\n",
|
||||
"<h3>Run training loop</h3>\n": "<h3>\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u30eb\u30fc\u30d7\u3092\u5b9f\u884c</h3>\n",
|
||||
"<h3>Sample data with current policy</h3>\n": "<h3>\u73fe\u5728\u306e\u30dd\u30ea\u30b7\u30fc\u3092\u542b\u3080\u30b5\u30f3\u30d7\u30eb\u30c7\u30fc\u30bf</h3>\n",
|
||||
"<h3>Train the model based on samples</h3>\n": "<h3>\u30b5\u30f3\u30d7\u30eb\u306b\u57fa\u3065\u3044\u3066\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b</h3>\n",
|
||||
"<h4>Configurations</h4>\n": "<h4>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</h4>\n",
|
||||
"<h4>Initialize</h4>\n": "<h4>[\u521d\u671f\u5316]</h4>\n",
|
||||
"<h4>Normalize advantage function</h4>\n": "<h4>\u30a2\u30c9\u30d0\u30f3\u30c6\u30fc\u30b8\u95a2\u6570\u306e\u6b63\u898f\u5316</h4>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> keeps track of the last observation from each worker, which is the input for the model to sample the next action </p>\n": "<p><span translate=no>_^_0_^_</span>\u5404\u30ef\u30fc\u30ab\u30fc\u304b\u3089\u306e\u6700\u5f8c\u306e\u89b3\u6e2c\u5024\u3092\u8ffd\u8de1\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u30e2\u30c7\u30eb\u304c\u6b21\u306e\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u5165\u529b\u3067\u3059</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> returns sampled from <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u304b\u3089\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u305f\u30ea\u30bf\u30fc\u30f3 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> are actions sampled from <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span><span translate=no>_^_1_^_</span>\u30a2\u30af\u30b7\u30e7\u30f3\u306f\u4ee5\u4e0b\u304b\u3089\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u307e\u3059 <span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is advantages sampled from <span translate=no>_^_2_^_</span>. Refer to sampling function in <a href=\"#main\">Main class</a> below for the calculation of <span translate=no>_^_3_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span>\u3001<span translate=no>_^_1_^_</span><span translate=no>_^_2_^_</span>\u5229\u70b9\u306f\u3069\u3053\u304b\u3089\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u3066\u3044\u308b\u306e\u304b\u3002\u306e\u8a08\u7b97\u306b\u3064\u3044\u3066\u306f\u3001<a href=\"#main\">\u4e0b\u8a18\u306e\u30e1\u30a4\u30f3\u30af\u30e9\u30b9\u306e\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u95a2\u6570\u3092\u53c2\u7167\u3057\u3066\u304f\u3060\u3055\u3044</a><span translate=no>_^_3_^_</span>\u3002</p>\n",
|
||||
"<p>A fully connected layer takes the flattened frame from third convolution layer, and outputs 512 features </p>\n": "<p>\u5b8c\u5168\u7d50\u5408\u5c64\u306f\u30013 \u756a\u76ee\u306e\u7573\u307f\u8fbc\u307f\u5c64\u304b\u3089\u5e73\u5766\u5316\u3055\u308c\u305f\u30d5\u30ec\u30fc\u30e0\u3092\u53d6\u308a\u51fa\u3057\u3001512 \u500b\u306e\u7279\u5fb4\u3092\u51fa\u529b\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>A fully connected layer to get logits for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30ed\u30b8\u30c3\u30c8\u3092\u53d6\u5f97\u3059\u308b\u305f\u3081\u306e\u5b8c\u5168\u63a5\u7d9a\u30ec\u30a4\u30e4\u30fc <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A fully connected layer to get value function </p>\n": "<p>\u30d0\u30ea\u30e5\u30fc\u95a2\u6570\u3092\u5f97\u308b\u305f\u3081\u306e\u5b8c\u5168\u9023\u7d50\u30ec\u30a4\u30e4\u30fc</p>\n",
|
||||
"<p>Add a new line to the screen periodically </p>\n": "<p>\u753b\u9762\u306b\u5b9a\u671f\u7684\u306b\u65b0\u3057\u3044\u884c\u3092\u8ffd\u52a0\u3057\u3066\u304f\u3060\u3055\u3044</p>\n",
|
||||
"<p>Add to tracker </p>\n": "<p>\u30c8\u30e9\u30c3\u30ab\u30fc\u306b\u8ffd\u52a0</p>\n",
|
||||
"<p>Calculate Entropy Bonus</p>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<p>\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u30dc\u30fc\u30ca\u30b9\u306e\u8a08\u7b97</p>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u52fe\u914d\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate policy loss </p>\n": "<p>\u4fdd\u967a\u5951\u7d04\u640d\u5931\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Calculate value function loss </p>\n": "<p>\u5024\u95a2\u6570\u640d\u5931\u306e\u8a08\u7b97</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u30af\u30ea\u30c3\u30d7\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Clipping range </p>\n": "<p>\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u7bc4\u56f2</p>\n",
|
||||
"<p>Configurations </p>\n": "<p>\u30b3\u30f3\u30d5\u30a3\u30ae\u30e5\u30ec\u30fc\u30b7\u30e7\u30f3</p>\n",
|
||||
"<p>Create the experiment </p>\n": "<p>\u5b9f\u9a13\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>Entropy bonus coefficient </p>\n": "<p>\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u30dc\u30fc\u30ca\u30b9\u4fc2\u6570</p>\n",
|
||||
"<p>GAE with <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>GATE (<span translate=no>_^_0_^_</span>\u304a\u3088\u3073\u4ed8\u304d) <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get value of after the final step </p>\n": "<p>\u6700\u5f8c\u306e\u30b9\u30c6\u30c3\u30d7\u306e\u5f8c\u306b\u5024\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>Initialize the trainer </p>\n": "<p>\u30c8\u30ec\u30fc\u30ca\u30fc\u3092\u521d\u671f\u5316</p>\n",
|
||||
"<p>It learns faster with a higher number of epochs, but becomes a little unstable; that is, the average episode reward does not monotonically increase over time. May be reducing the clipping range might solve it. </p>\n": "<p>\u30a8\u30dd\u30c3\u30af\u6570\u304c\u591a\u3044\u307b\u3069\u5b66\u7fd2\u306f\u901f\u304f\u306a\u308a\u307e\u3059\u304c\u3001\u5c11\u3057\u4e0d\u5b89\u5b9a\u306b\u306a\u308a\u307e\u3059\u3002\u3064\u307e\u308a\u3001\u30a8\u30d4\u30bd\u30fc\u30c9\u306e\u5e73\u5747\u5831\u916c\u306f\u6642\u9593\u306e\u7d4c\u904e\u3068\u3068\u3082\u306b\u5358\u8abf\u306b\u5897\u52a0\u3057\u307e\u305b\u3093\u3002\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u7bc4\u56f2\u3092\u72ed\u304f\u3059\u308b\u3053\u3068\u3067\u89e3\u6c7a\u3059\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\u3002</p>\n",
|
||||
"<p>Learning rate </p>\n": "<p>\u5b66\u7fd2\u7387</p>\n",
|
||||
"<p>Number of mini batches </p>\n": "<p>\u30df\u30cb\u30d0\u30c3\u30c1\u6570</p>\n",
|
||||
"<p>Number of steps to run on each process for a single update </p>\n": "<p>1 \u56de\u306e\u66f4\u65b0\u3067\u5404\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3059\u308b\u30b9\u30c6\u30c3\u30d7\u306e\u6570</p>\n",
|
||||
"<p>Number of updates </p>\n": "<p>\u66f4\u65b0\u56de\u6570</p>\n",
|
||||
"<p>Number of worker processes </p>\n": "<p>\u30ef\u30fc\u30ab\u30fc\u30d7\u30ed\u30bb\u30b9\u306e\u6570</p>\n",
|
||||
"<p>PPO Loss </p>\n": "<p>PPO \u30ed\u30b9</p>\n",
|
||||
"<p>Run and monitor the experiment </p>\n": "<p>\u5b9f\u9a13\u306e\u5b9f\u884c\u3068\u76e3\u8996</p>\n",
|
||||
"<p>Sampled observations are fed into the model to get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>; we are treating observations as state </p>\n": "<p><span translate=no>_^_0_^_</span>\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u3055\u308c\u305f\u89b3\u6e2c\u5024\u306f\u30e2\u30c7\u30eb\u306b\u5165\u529b\u3055\u308c\u3001\u53d6\u5f97\u3055\u308c\u307e\u3059<span translate=no>_^_1_^_</span>\u3002\u89b3\u6e2c\u5024\u306f\u72b6\u614b\u3068\u3057\u3066\u6271\u3044\u307e\u3059</p>\n",
|
||||
"<p>Save tracked indicators. </p>\n": "<p>\u8ffd\u8de1\u6307\u6a19\u3092\u4fdd\u5b58\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Scale observations from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u89b3\u6e2c\u5024\u3092\u304b\u3089\u306b\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Select device </p>\n": "<p>\u30c7\u30d0\u30a4\u30b9\u3092\u9078\u629e</p>\n",
|
||||
"<p>Set learning rate </p>\n": "<p>\u5b66\u7fd2\u7387\u3092\u8a2d\u5b9a</p>\n",
|
||||
"<p>Stop the workers </p>\n": "<p>\u52b4\u50cd\u8005\u3092\u6b62\u3081\u308d</p>\n",
|
||||
"<p>The first convolution layer takes a 84x84 frame and produces a 20x20 frame </p>\n": "<p>\u6700\u521d\u306e\u7573\u307f\u8fbc\u307f\u5c64\u306f 84 x 84 \u30d5\u30ec\u30fc\u30e0\u3067\u300120 x 20 \u30d5\u30ec\u30fc\u30e0\u3092\u751f\u6210\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>The second convolution layer takes a 20x20 frame and produces a 9x9 frame </p>\n": "<p>2 \u756a\u76ee\u306e\u7573\u307f\u8fbc\u307f\u5c64\u306f 20x20 \u30d5\u30ec\u30fc\u30e0\u3067\u30019x9 \u30d5\u30ec\u30fc\u30e0\u3092\u751f\u6210\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>The third convolution layer takes a 9x9 frame and produces a 7x7 frame </p>\n": "<p>3 \u756a\u76ee\u306e\u7573\u307f\u8fbc\u307f\u5c64\u306f 9x9 \u30d5\u30ec\u30fc\u30e0\u3067 7x7 \u30d5\u30ec\u30fc\u30e0\u3092\u751f\u6210\u3057\u307e\u3059\u3002</p>\n",
|
||||
"<p>Update parameters based on gradients </p>\n": "<p>\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u306b\u57fa\u3065\u3044\u3066\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u66f4\u65b0</p>\n",
|
||||
"<p>Value Loss </p>\n": "<p>\u4fa1\u5024\u640d\u5931</p>\n",
|
||||
"<p>Value loss coefficient </p>\n": "<p>\u4fa1\u5024\u640d\u5931\u4fc2\u6570</p>\n",
|
||||
"<p>You can change this while the experiment is running. \u2699\ufe0f Learning rate. </p>\n": "<p>\u30c6\u30b9\u30c8\u306e\u5b9f\u884c\u4e2d\u306b\u3053\u308c\u3092\u5909\u66f4\u3067\u304d\u307e\u3059\u3002\u2699\ufe0f \u5b66\u7fd2\u7387\u3002</p>\n",
|
||||
"<p>Zero out the previously calculated gradients </p>\n": "<p>\u4ee5\u524d\u306b\u8a08\u7b97\u3057\u305f\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u3092\u30bc\u30ed\u306b\u3057\u307e\u3059</p>\n",
|
||||
"<p>calculate advantages </p>\n": "<p>\u5229\u70b9\u3092\u8a08\u7b97</p>\n",
|
||||
"<p>collect episode info, which is available if an episode finished; this includes total reward and length of the episode - look at <span translate=no>_^_0_^_</span> to see how it works. </p>\n": "<p>\u30a8\u30d4\u30bd\u30fc\u30c9\u306e\u60c5\u5831\u3092\u96c6\u3081\u307e\u3057\u3087\u3046\u3002<span translate=no>_^_0_^_</span>\u30a8\u30d4\u30bd\u30fc\u30c9\u304c\u7d42\u4e86\u3057\u305f\u3068\u304d\u306b\u5165\u624b\u3067\u304d\u307e\u3059\u3002\u3053\u308c\u306b\u306f\u5831\u916c\u7dcf\u984d\u3084\u30a8\u30d4\u30bd\u30fc\u30c9\u306e\u9577\u3055\u304c\u542b\u307e\u308c\u307e\u3059\u3002\u4ed5\u7d44\u307f\u3092\u78ba\u8a8d\u3057\u3066\u307f\u307e\u3057\u3087\u3046\u3002</p>\n",
|
||||
"<p>create workers </p>\n": "<p>\u30ef\u30fc\u30ab\u30fc\u3092\u4f5c\u6210</p>\n",
|
||||
"<p>for each mini batch </p>\n": "<p>\u5404\u30df\u30cb\u30d0\u30c3\u30c1\u7528</p>\n",
|
||||
"<p>for monitoring </p>\n": "<p>\u76e3\u8996\u7528</p>\n",
|
||||
"<p>get mini batch </p>\n": "<p>\u30df\u30cb\u30d0\u30c3\u30c1\u3092\u5165\u624b</p>\n",
|
||||
"<p>get results after executing the actions </p>\n": "<p>\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c\u3057\u305f\u5f8c\u306b\u7d50\u679c\u3092\u53d6\u5f97</p>\n",
|
||||
"<p>initialize tensors for observations </p>\n": "<p>\u89b3\u6e2c\u7528\u306e\u30c6\u30f3\u30bd\u30eb\u3092\u521d\u671f\u5316</p>\n",
|
||||
"<p>last 100 episode information </p>\n": "<p>\u6700\u5f8c\u306e 100 \u8a71\u306e\u60c5\u5831</p>\n",
|
||||
"<p>model </p>\n": "<p>\u30e2\u30c7\u30eb</p>\n",
|
||||
"<p>number of epochs to train the model with sampled data </p>\n": "<p>\u30b5\u30f3\u30d7\u30eb\u30c7\u30fc\u30bf\u3092\u4f7f\u7528\u3057\u3066\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u30a8\u30dd\u30c3\u30af\u306e\u6570</p>\n",
|
||||
"<p>number of mini batches </p>\n": "<p>\u30df\u30cb\u30d0\u30c3\u30c1\u6570</p>\n",
|
||||
"<p>number of steps to run on each process for a single update </p>\n": "<p>1 \u56de\u306e\u66f4\u65b0\u3067\u5404\u30d7\u30ed\u30bb\u30b9\u3067\u5b9f\u884c\u3059\u308b\u30b9\u30c6\u30c3\u30d7\u306e\u6570</p>\n",
|
||||
"<p>number of updates </p>\n": "<p>\u66f4\u65b0\u56de\u6570</p>\n",
|
||||
"<p>number of worker processes </p>\n": "<p>\u30ef\u30fc\u30ab\u30fc\u30d7\u30ed\u30bb\u30b9\u306e\u6570</p>\n",
|
||||
"<p>optimizer </p>\n": "<p>\u30aa\u30d7\u30c6\u30a3\u30de\u30a4\u30b6\u30fc</p>\n",
|
||||
"<p>run sampled actions on each worker </p>\n": "<p>\u5404\u30ef\u30fc\u30ab\u30fc\u3067\u30b5\u30f3\u30d7\u30eb\u30a2\u30af\u30b7\u30e7\u30f3\u3092\u5b9f\u884c</p>\n",
|
||||
"<p>sample <span translate=no>_^_0_^_</span> from each worker </p>\n": "<p><span translate=no>_^_0_^_</span>\u5404\u52b4\u50cd\u8005\u304b\u3089\u306e\u30b5\u30f3\u30d7\u30eb</p>\n",
|
||||
"<p>sample actions from <span translate=no>_^_0_^_</span> for each worker; this returns arrays of size <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u5404\u30ef\u30fc\u30ab\u30fc\u306e\u30b5\u30f3\u30d7\u30eb\u30a2\u30af\u30b7\u30e7\u30f3\u3002\u3053\u308c\u306f\u30b5\u30a4\u30ba\u306e\u914d\u5217\u3092\u8fd4\u3057\u307e\u3059 <span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>sample with current policy </p>\n": "<p>\u73fe\u884c\u30dd\u30ea\u30b7\u30fc\u306e\u30b5\u30f3\u30d7\u30eb</p>\n",
|
||||
"<p>samples are currently in <span translate=no>_^_0_^_</span> table, we should flatten it for training </p>\n": "<p><span translate=no>_^_0_^_</span>\u30b5\u30f3\u30d7\u30eb\u306f\u73fe\u5728\u30c6\u30fc\u30d6\u30eb\u306b\u3042\u308b\u306e\u3067\u3001\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u7528\u306b\u5e73\u3089\u306b\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059</p>\n",
|
||||
"<p>shuffle for each epoch </p>\n": "<p>\u5404\u30a8\u30dd\u30c3\u30af\u306e\u30b7\u30e3\u30c3\u30d5\u30eb</p>\n",
|
||||
"<p>size of a mini batch </p>\n": "<p>\u30df\u30cb\u30d0\u30c3\u30c1\u306e\u30b5\u30a4\u30ba</p>\n",
|
||||
"<p>total number of samples for a single update </p>\n": "<p>1 \u56de\u306e\u66f4\u65b0\u3067\u306e\u30b5\u30f3\u30d7\u30eb\u306e\u7dcf\u6570</p>\n",
|
||||
"<p>train </p>\n": "<p>\u5217\u8eca</p>\n",
|
||||
"<p>train the model </p>\n": "<p>\u30e2\u30c7\u30eb\u306e\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0</p>\n",
|
||||
"<p>\u2699\ufe0f Clip range. </p>\n": "<p>\u2699\ufe0f \u30af\u30ea\u30c3\u30d7\u30ec\u30f3\u30b8\u3002</p>\n",
|
||||
"<p>\u2699\ufe0f Entropy bonus coefficient. You can change this while the experiment is running. </p>\n": "<p>\u2699\ufe0f \u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u30dc\u30fc\u30ca\u30b9\u4fc2\u6570\u3002\u3053\u308c\u306f\u5b9f\u9a13\u306e\u5b9f\u884c\u4e2d\u306b\u5909\u66f4\u3067\u304d\u307e\u3059\u3002</p>\n",
|
||||
"<p>\u2699\ufe0f Number of epochs to train the model with sampled data. You can change this while the experiment is running. </p>\n": "<p>\u2699\ufe0f \u30b5\u30f3\u30d7\u30eb\u30c7\u30fc\u30bf\u3092\u4f7f\u7528\u3057\u3066\u30e2\u30c7\u30eb\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u30a8\u30dd\u30c3\u30af\u306e\u6570\u3002\u3053\u308c\u306f\u5b9f\u9a13\u306e\u5b9f\u884c\u4e2d\u306b\u5909\u66f4\u3067\u304d\u307e\u3059\u3002</p>\n",
|
||||
"<p>\u2699\ufe0f Value loss coefficient. You can change this while the experiment is running. </p>\n": "<p>\u2699\ufe0f \u4fa1\u5024\u640d\u5931\u4fc2\u6570\u3002\u3053\u308c\u306f\u5b9f\u9a13\u306e\u5b9f\u884c\u4e2d\u306b\u5909\u66f4\u3067\u304d\u307e\u3059\u3002</p>\n",
|
||||
"Annotated implementation to train a PPO agent on Atari Breakout game.": "Atari Breakout \u30b2\u30fc\u30e0\u3067 PPO \u30a8\u30fc\u30b8\u30a7\u30f3\u30c8\u3092\u30c8\u30ec\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u305f\u3081\u306e\u6ce8\u91c8\u4ed8\u304d\u5b9f\u88c5\u3002",
|
||||
"PPO Experiment with Atari Breakout": "\u30a2\u30bf\u30ea\u30fb\u30d6\u30ec\u30a4\u30af\u30a2\u30a6\u30c8\u306b\u3088\u308bPPO\u5b9f\u9a13"
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"<h1>PPO Experiment with Atari Breakout</h1>\n<p>This experiment trains Proximal Policy Optimization (PPO) agent Atari Breakout game on OpenAI Gym. It runs the <a href=\"../game.html\">game environments on multiple processes</a> to sample efficiently.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f\"><span translate=no>_^_1_^_</span></a></p>\n": "<h1>\u0d85\u0da7\u0dcf\u0dbb\u0dd2\u0d9a\u0da9\u0dcf\u0dc0\u0dd0\u0da7\u0dd3\u0db8 \u0dc3\u0db8\u0d9f PPO \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8</h1>\n<p>\u0db8\u0dd9\u0db8\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 OpenAI Gym \u0dc4\u0dd2 \u0db4\u0dca\u0dbb\u0ddc\u0d9a\u0dca\u0dc3\u0dd2\u0db8\u0dbd\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0dad\u0dca\u0dad\u0dd2 \u0db4\u0dca\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0dd2\u0d9a\u0dbb\u0dab\u0dba (PPO) \u0db1\u0dd2\u0dba\u0ddd\u0da2\u0dd2\u0dad \u0d85\u0da7\u0dcf\u0dbb\u0dd2 \u0db6\u0dca\u0dbb\u0dda\u0d9a\u0dca \u0d9a\u0dca\u0dbb\u0dd3\u0da9\u0dcf\u0dc0 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0dba\u0dd2. \u0d9a\u0dcf\u0dbb\u0dca\u0dba\u0d9a\u0dca\u0dc2\u0db8\u0dc0 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0dba <a href=\"../game.html\">\u0db6\u0dc4\u0dd4 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0\u0dbd\u0dd3\u0db1\u0dca\u0dc4\u0dd2 \u0d9a\u0dca\u0dbb\u0dd3\u0da9\u0dcf \u0db4\u0dbb\u0dd2\u0dc3\u0dbb\u0dba\u0db1\u0dca</a> \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0dba\u0dd2. </p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f\"> <span translate=no>_^_1_^_</span></a></p>\n",
|
||||
"<h2>Model</h2>\n": "<h2>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba</h2>\n",
|
||||
"<h2>Run it</h2>\n": "<h2>\u0d91\u0dba\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dbb\u0db1\u0dca\u0db1</h2>\n",
|
||||
"<h2>Trainer</h2>\n": "<h2>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4</h2>\n",
|
||||
"<h3>Calculate total loss</h3>\n": "<h3>\u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab\u0d85\u0dbd\u0dcf\u0db7\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Destroy</h3>\n<p>Stop the workers</p>\n": "<h3>\u0dc0\u0dd2\u0db1\u0dcf\u0dc1\u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<p>\u0d9a\u0db8\u0dca\u0d9a\u0dbb\u0dd4\u0dc0\u0db1\u0dca\u0db1\u0dc0\u0dad\u0dca\u0dc0\u0db1\u0dca\u0db1</p>\n",
|
||||
"<h3>Run training loop</h3>\n": "<h3>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dbd\u0dd6\u0db4\u0dba \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h3>Sample data with current policy</h3>\n": "<h3>\u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0dad\u0dca\u0dad\u0dd2\u0dba \u0dc3\u0db8\u0d9f \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0daf\u0dad\u0dca\u0dad</h3>\n",
|
||||
"<h3>Train the model based on samples</h3>\n": "<h3>\u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd\u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca\u0dc0 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n",
|
||||
"<h4>Configurations</h4>\n": "<h4>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca</h4>\n",
|
||||
"<h4>Initialize</h4>\n": "<h4>\u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a\u0dbb\u0db1\u0dca\u0db1</h4>\n",
|
||||
"<h4>Normalize advantage function</h4>\n": "<h4>\u0dc0\u0dcf\u0dc3\u0dd2\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0d9a\u0dcf\u0dbb\u0dd2\u0dad\u0dca\u0dc0\u0dba \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1</h4>\n",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> keeps track of the last observation from each worker, which is the input for the model to sample the next action </p>\n": "<p><span translate=no>_^_0_^_</span> \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0dc3\u0dda\u0dc0\u0d9a\u0dba\u0dcf\u0d9c\u0dd9\u0db1\u0dca \u0d85\u0dc0\u0dc3\u0dcf\u0db1 \u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0d9a\u0dbb\u0dba\u0dd2, \u0d91\u0dba \u0d8a\u0dc5\u0d9f \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0 \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0da7 \u0d86\u0daf\u0dcf\u0db1\u0dba \u0dc0\u0dda </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> returns sampled from <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> \u0dc3\u0dd2\u0da7 \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0dcf\u0db7 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> are actions sampled from <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>, \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf <span translate=no>_^_1_^_</span> \u0dc0\u0dbd\u0dd2\u0db1\u0dca \u0db1\u0dd2\u0dba\u0dd0\u0dbd\u0dd3 \u0d87\u0dad <span translate=no>_^_2_^_</span> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is advantages sampled from <span translate=no>_^_2_^_</span>. Refer to sampling function in <a href=\"#main\">Main class</a> below for the calculation of <span translate=no>_^_3_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span>, \u0d9a\u0ddc\u0dc4\u0dd9\u0db1\u0dca\u0daf? <span translate=no>_^_1_^_</span> \u0dc0\u0dcf\u0dc3\u0dd2 \u0dbd\u0db6\u0dcf \u0d87\u0dad <span translate=no>_^_2_^_</span>. \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dc4\u0dad <a href=\"#main\">\u0db4\u0dca\u0dbb\u0db0\u0dcf\u0db1 \u0db4\u0db1\u0dca\u0dad\u0dd2\u0dba\u0dda</a> \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0dc1\u0dca\u0dbb\u0dd2\u0dad\u0dba \u0dc0\u0dd9\u0dad \u0dba\u0ddc\u0db8\u0dd4 \u0dc0\u0db1\u0dca\u0db1 <span translate=no>_^_3_^_</span>. </p>\n",
|
||||
"<p>A fully connected layer takes the flattened frame from third convolution layer, and outputs 512 features </p>\n": "<p>\u0dc3\u0db8\u0dca\u0db4\u0dd4\u0dbb\u0dca\u0dab\u0dba\u0dd9\u0db1\u0dca\u0db8\u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dad\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0d9a\u0dca \u0db4\u0dd0\u0dad\u0dbd\u0dd2 \u0dbb\u0dcf\u0db8\u0dd4\u0dc0 \u0dad\u0dd9\u0dc0\u0db1 \u0d9a\u0dd0\u0da7\u0dd2 \u0d9c\u0dd0\u0dc3\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dae\u0dbb\u0dba\u0dd9\u0db1\u0dca \u0d9c\u0dd9\u0db1 \u0dba\u0db1 \u0d85\u0dad\u0dbb \u0dc0\u0dd2\u0dc1\u0dda\u0dc2\u0dcf\u0d82\u0d9c 512 \u0d9a\u0dca \u0db4\u0dca\u0dbb\u0dad\u0dd2\u0daf\u0dcf\u0db1\u0dba \u0d9a\u0dbb\u0dba\u0dd2 </p>\n",
|
||||
"<p>A fully connected layer to get logits for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0dc3\u0db3\u0dc4\u0dcf\u0db4\u0dd2\u0dc0\u0dd2\u0dc3\u0dd4\u0db8\u0dca \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dad\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0d9a\u0dca <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>A fully connected layer to get value function </p>\n": "<p>\u0d85\u0d9c\u0dba\u0dc1\u0dca\u0dbb\u0dd2\u0dad\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc3\u0db8\u0dca\u0db6\u0db1\u0dca\u0db0\u0dd2\u0dad \u0dad\u0da7\u0dca\u0da7\u0dd4\u0dc0\u0d9a\u0dca </p>\n",
|
||||
"<p>Add a new line to the screen periodically </p>\n": "<p>\u0dc0\u0dbb\u0dd2\u0db1\u0dca\u0dc0\u0dbb \u0dad\u0dd2\u0dbb\u0dba\u0da7 \u0db1\u0dc0 \u0dbb\u0dda\u0d9b\u0dcf\u0dc0\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Add to tracker </p>\n": "<p>\u0da7\u0dca\u0dbb\u0dd0\u0d9a\u0dbb\u0dca\u0dc0\u0dd9\u0dad \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate Entropy Bonus</p>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<p>\u0d91\u0db1\u0dca\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd2\u0db6\u0ddd\u0db1\u0dc3\u0dca \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1</p>\n<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate policy loss </p>\n": "<p>\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0dad\u0dca\u0dad\u0dd2\u0d85\u0dbd\u0dcf\u0db7\u0dba \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Calculate value function loss </p>\n": "<p>\u0d85\u0d9c\u0dba\u0dc1\u0dca\u0dbb\u0dd2\u0dad\u0dba \u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0d9c\u0dab\u0db1\u0dba </p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dca\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a </p>\n",
|
||||
"<p>Clipping range </p>\n": "<p>\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dd2\u0db1\u0dca\u0db4\u0dbb\u0dcf\u0dc3\u0dba </p>\n",
|
||||
"<p>Configurations </p>\n": "<p>\u0dc0\u0dd2\u0db1\u0dca\u0dba\u0dcf\u0dc3\u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dca </p>\n",
|
||||
"<p>Create the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0dc3\u0dcf\u0daf\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Entropy bonus coefficient </p>\n": "<p>\u0d91\u0db1\u0dca\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd2\u0db4\u0dca\u0dbb\u0dc3\u0dcf\u0daf \u0dc3\u0d82\u0d9c\u0dd4\u0dab\u0d9a\u0dba </p>\n",
|
||||
"<p>GAE with <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>GAE\u0dc3\u0db8\u0d9f <span translate=no>_^_0_^_</span> \u0dc3\u0dc4 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Get value of after the final step </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0dcf\u0db1\u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0dc0\u0da7\u0dd2\u0db1\u0dcf\u0d9a\u0db8 \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Initialize the trainer </p>\n": "<p>\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0d9a\u0dbb\u0dd4\u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>It learns faster with a higher number of epochs, but becomes a little unstable; that is, the average episode reward does not monotonically increase over time. May be reducing the clipping range might solve it. </p>\n": "<p>\u0d91\u0dba\u0dc0\u0dd0\u0da9\u0dd2 \u0d91\u0db4\u0ddc\u0da0\u0dca \u0dc3\u0d82\u0d9b\u0dca\u0dba\u0dcf\u0dc0\u0d9a\u0dca \u0dc3\u0db8\u0d9f \u0dc0\u0dda\u0d9c\u0dba\u0dd9\u0db1\u0dca \u0d89\u0d9c\u0dd9\u0db1 \u0d9c\u0db1\u0dd3, \u0db1\u0db8\u0dd4\u0dad\u0dca \u0da7\u0dd2\u0d9a\u0d9a\u0dca \u0d85\u0dc3\u0dca\u0dae\u0dcf\u0dba\u0dd3 \u0dc0\u0dda; \u0d91\u0db1\u0db8\u0dca, \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0d9a\u0dae\u0dcf\u0d82\u0d9c \u0dc0\u0dd2\u0db4\u0dcf\u0d9a\u0dba \u0d9a\u0dcf\u0dbd\u0dba\u0dad\u0dca \u0dc3\u0db8\u0d9f \u0d92\u0d9a\u0dcf\u0d9a\u0dcf\u0dbb\u0dd3 \u0dbd\u0dd9\u0dc3 \u0dc0\u0dd0\u0da9\u0dd2 \u0db1\u0ddc\u0dc0\u0dda. \u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dd2\u0db1\u0dca \u0db4\u0dbb\u0dcf\u0dc3\u0dba \u0d85\u0da9\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0d91\u0dba \u0dc0\u0dd2\u0dc3\u0db3\u0dd2\u0dba \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. </p>\n",
|
||||
"<p>Learning rate </p>\n": "<p>\u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba </p>\n",
|
||||
"<p>Number of mini batches </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of steps to run on each process for a single update </p>\n": "<p>\u0dad\u0db1\u0dd2\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0\u0dbd\u0dd2\u0dba \u0db8\u0dad \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of updates </p>\n": "<p>\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1\u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>Number of worker processes </p>\n": "<p>\u0dc3\u0dda\u0dc0\u0d9a\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0\u0dbd\u0dd2 \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>PPO Loss </p>\n": "<p>PPO\u0db4\u0dcf\u0da9\u0dd4\u0dc0 </p>\n",
|
||||
"<p>Run and monitor the experiment </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0db0\u0dcf\u0dc0\u0db1\u0dba \u0d9a\u0dbb \u0d85\u0db0\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Sampled observations are fed into the model to get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>; we are treating observations as state </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab \u0dbd\u0db6\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0da7 \u0db4\u0ddd\u0dc2\u0dab\u0dba \u0dc0\u0db1 <span translate=no>_^_0_^_</span> \u0d85\u0dad\u0dbb <span translate=no>_^_1_^_</span>; \u0d85\u0db4\u0dd2 \u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab \u0dbb\u0dcf\u0da2\u0dca\u0dba \u0dbd\u0dd9\u0dc3 \u0dc3\u0dbd\u0d9a\u0db8\u0dd4 </p>\n",
|
||||
"<p>Save tracked indicators. </p>\n": "<p>\u0dbd\u0dd4\u0dc4\u0dd4\u0db6\u0dd0\u0db3\u0d87\u0dad\u0dd2 \u0daf\u0dbb\u0dca\u0dc1\u0d9a \u0dc3\u0dd4\u0dbb\u0d9a\u0dd2\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>Scale observations from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0dc3\u0dd2\u0da7 <span translate=no>_^_0_^_</span> \u0db4\u0dbb\u0dd2\u0db8\u0dcf\u0dab \u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>Select device </p>\n": "<p>\u0d8b\u0db4\u0dcf\u0d82\u0d9c\u0dba\u0dad\u0ddd\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Set learning rate </p>\n": "<p>\u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca\u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba \u0dc3\u0d9a\u0dc3\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Stop the workers </p>\n": "<p>\u0d9a\u0db8\u0dca\u0d9a\u0dbb\u0dd4\u0dc0\u0db1\u0dca\u0db1\u0dc0\u0dad\u0dca\u0dc0\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>The first convolution layer takes a 84x84 frame and produces a 20x20 frame </p>\n": "<p>\u0db4\u0dc5\u0db8\u0dd4\u0d9a\u0dd0\u0da7\u0dd2 \u0d9c\u0dd0\u0dc3\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dad\u0dbb\u0dba 84x84 \u0dbb\u0dcf\u0db8\u0dd4\u0dc0\u0d9a\u0dca \u0d9c\u0dd9\u0db1 20x20 \u0dbb\u0dcf\u0db8\u0dd4\u0dc0\u0d9a\u0dca \u0db1\u0dd2\u0db4\u0daf\u0dc0\u0dba\u0dd2 </p>\n",
|
||||
"<p>The second convolution layer takes a 20x20 frame and produces a 9x9 frame </p>\n": "<p>\u0daf\u0dd9\u0dc0\u0db1\u0d9a\u0dd0\u0da7\u0dd2 \u0d9c\u0dd0\u0dc3\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dad\u0dbb\u0dba 20x20 \u0dbb\u0dcf\u0db8\u0dd4\u0dc0\u0d9a\u0dca \u0d9c\u0dd9\u0db1 9x9 \u0dbb\u0dcf\u0db8\u0dd4\u0dc0\u0d9a\u0dca \u0db1\u0dd2\u0db4\u0daf\u0dc0\u0dba\u0dd2 </p>\n",
|
||||
"<p>The third convolution layer takes a 9x9 frame and produces a 7x7 frame </p>\n": "<p>\u0dad\u0dd9\u0dc0\u0db1\u0d9a\u0dd0\u0da7\u0dd2 \u0d9c\u0dd0\u0dc3\u0dd4\u0dab\u0dd4 \u0dc3\u0dca\u0dad\u0dbb\u0dba 9x9 \u0dbb\u0dcf\u0db8\u0dd4\u0dc0\u0d9a\u0dca \u0d9c\u0dd9\u0db1 7x7 \u0dbb\u0dcf\u0db8\u0dd4\u0dc0\u0d9a\u0dca \u0db1\u0dd2\u0db4\u0daf\u0dc0\u0dba\u0dd2 </p>\n",
|
||||
"<p>Update parameters based on gradients </p>\n": "<p>\u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a\u0db8\u0dad \u0db4\u0daf\u0db1\u0db8\u0dca\u0dc0 \u0db4\u0dbb\u0dcf\u0db8\u0dd2\u0dad\u0dd3\u0db1\u0dca \u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>Value Loss </p>\n": "<p>\u0d85\u0d9c\u0dba\u0db1\u0dd0\u0dad\u0dd2\u0dc0\u0dd3\u0db8 </p>\n",
|
||||
"<p>Value loss coefficient </p>\n": "<p>\u0d85\u0d9c\u0dba\u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0dc3\u0d82\u0d9c\u0dd4\u0dab\u0d9a\u0dba </p>\n",
|
||||
"<p>You can change this while the experiment is running. <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f/hyper_params\"><span translate=no>_^_0_^_</span></a> \u2699\ufe0f Learning rate. </p>\n": "<p>\u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf\u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc0\u0db1 \u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0d94\u0db6\u0da7 \u0db8\u0dd9\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f/hyper_params\"><span translate=no>_^_0_^_</span></a> \u2699\ufe0f \u0d89\u0d9c\u0dd9\u0db1\u0dd4\u0db8\u0dca \u0d85\u0db1\u0dd4\u0db4\u0dcf\u0dad\u0dba. </p>\n",
|
||||
"<p>Zero out the previously calculated gradients </p>\n": "<p>\u0d9a\u0dbd\u0dd2\u0db1\u0dca\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0d85\u0db1\u0dd4\u0d9a\u0dca\u0dbb\u0db8\u0dd2\u0d9a \u0dc1\u0dd4\u0db1\u0dca\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 </p>\n",
|
||||
"<p>calculate advantages </p>\n": "<p>\u0dc0\u0dcf\u0dc3\u0dd2\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>collect episode info, which is available if an episode finished; this includes total reward and length of the episode - look at <span translate=no>_^_0_^_</span> to see how it works. </p>\n": "<p>\u0d9a\u0dae\u0dcf\u0d82\u0d9c\u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1, \u0d9a\u0dae\u0dcf\u0d82\u0d9c\u0dba\u0d9a\u0dca \u0d85\u0dc0\u0dc3\u0db1\u0dca \u0dc0\u0dd4\u0dc0\u0dc4\u0ddc\u0dad\u0dca \u0d91\u0dba \u0dbd\u0db6\u0dcf \u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2\u0dba; \u0db8\u0dd9\u0dba\u0da7 \u0d9a\u0dae\u0dcf\u0d82\u0d9c\u0dba\u0dda \u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc0\u0dd2\u0db4\u0dcf\u0d9a\u0dba \u0dc3\u0dc4 \u0daf\u0dd2\u0d9c \u0d87\u0dad\u0dd4\u0dc5\u0dad\u0dca \u0dc0\u0dda - \u0d91\u0dba \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc0\u0db1 \u0d86\u0d9a\u0dcf\u0dbb\u0dba <span translate=no>_^_0_^_</span> \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0da7 \u0db6\u0dbd\u0db1\u0dca\u0db1. </p>\n",
|
||||
"<p>create workers </p>\n": "<p>\u0d9a\u0db8\u0dca\u0d9a\u0dbb\u0dd4\u0dc0\u0db1\u0dca\u0db1\u0dd2\u0dbb\u0dca\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>for each mini batch </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d9a\u0dd4\u0da9\u0dcf \u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8 \u0dc3\u0db3\u0dc4\u0dcf </p>\n",
|
||||
"<p>for monitoring </p>\n": "<p>\u0d85\u0db0\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dba\u0dc3\u0db3\u0dc4\u0dcf </p>\n",
|
||||
"<p>get mini batch </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8 \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>get results after executing the actions </p>\n": "<p>\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0\u0db1\u0dca\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0dd9\u0db1\u0dca \u0db4\u0dc3\u0dd4 \u0db4\u0dca\u0dbb\u0dad\u0dd2 results \u0dbd \u0dbd\u0db6\u0dcf \u0d9c\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>initialize tensors for observations </p>\n": "<p>\u0db1\u0dd2\u0dbb\u0dd3\u0d9a\u0dca\u0dc2\u0dab\u0dc3\u0db3\u0dc4\u0dcf \u0d86\u0dad\u0dad\u0dd3\u0db1\u0dca \u0d86\u0dbb\u0db8\u0dca\u0db7 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>last 100 episode information </p>\n": "<p>\u0d85\u0dc0\u0dc3\u0db1\u0dca100 \u0d9a\u0dae\u0dcf\u0d82\u0d9c \u0dad\u0ddc\u0dbb\u0dad\u0dd4\u0dbb\u0dd4 </p>\n",
|
||||
"<p>model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba </p>\n",
|
||||
"<p>number of epochs to train the model with sampled data </p>\n": "<p>\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0d9f \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0db4\u0ddc\u0da0\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>number of mini batches </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0dca \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>number of steps to run on each process for a single update </p>\n": "<p>\u0dad\u0db1\u0dd2\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0d9a\u0dca \u0d91\u0d9a\u0dca \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0\u0dbd\u0dd2\u0dba \u0db8\u0dad \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0da7 \u0db4\u0dd2\u0dba\u0dc0\u0dbb \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>number of updates </p>\n": "<p>\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1\u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>number of worker processes </p>\n": "<p>\u0dc3\u0dda\u0dc0\u0d9a\u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dc0\u0dbd\u0dd2 \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>optimizer </p>\n": "<p>\u0db4\u0dca\u200d\u0dbb\u0dc1\u0dc3\u0dca\u0dad\u0d9a\u0dbb\u0dab\u0dba </p>\n",
|
||||
"<p>run sampled actions on each worker </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dda\u0dc0\u0d9a\u0dba\u0dcf \u0db8\u0dad \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a </p>\n",
|
||||
"<p>sample <span translate=no>_^_0_^_</span> from each worker </p>\n": "<p>\u0dc3\u0dd1\u0db8 <span translate=no>_^_0_^_</span> \u0dc3\u0dda\u0dc0\u0d9a\u0dba\u0dd9\u0d9a\u0dd4\u0d9c\u0dda\u0db8 \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba </p>\n",
|
||||
"<p>sample actions from <span translate=no>_^_0_^_</span> for each worker; this returns arrays of size <span translate=no>_^_1_^_</span> </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0dc3\u0dda\u0dc0\u0d9a\u0dba\u0dcf <span translate=no>_^_0_^_</span> \u0dc3\u0db3\u0dc4\u0dcf \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf; \u0db8\u0dd9\u0dba \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba\u0dda \u0d85\u0dbb\u0dcf \u0db1\u0dd0\u0dc0\u0dad \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0dba\u0dd2 <span translate=no>_^_1_^_</span> </p>\n",
|
||||
"<p>sample with current policy </p>\n": "<p>\u0dc0\u0dad\u0dca\u0db8\u0db1\u0dca\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0db4\u0dad\u0dca\u0dad\u0dd2\u0dba \u0dc3\u0db8\u0d9f \u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2\u0dba </p>\n",
|
||||
"<p>samples are currently in <span translate=no>_^_0_^_</span> table, we should flatten it for training </p>\n": "<p>\u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd\u0daf\u0dd0\u0db1\u0da7 <span translate=no>_^_0_^_</span> \u0dc0\u0d9c\u0dd4\u0dc0\u0dda \u0d87\u0dad, \u0d85\u0db4\u0dd2 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4\u0dc0 \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0dba \u0dc3\u0db8\u0dad\u0dbd\u0dcf \u0d9a\u0dc5 \u0dba\u0dd4\u0dad\u0dd4\u0dba </p>\n",
|
||||
"<p>shuffle for each epoch </p>\n": "<p>\u0d91\u0d9a\u0dca\u0d91\u0d9a\u0dca \u0d8a\u0db4\u0ddd\u0da0\u0dca \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dbd\u0dc0\u0db8\u0dca \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>size of a mini batch </p>\n": "<p>\u0d9a\u0dd4\u0da9\u0dcf\u0d9a\u0dab\u0dca\u0da9\u0dcf\u0dba\u0db8\u0d9a \u0db4\u0dca\u0dbb\u0db8\u0dcf\u0dab\u0dba </p>\n",
|
||||
"<p>total number of samples for a single update </p>\n": "<p>\u0dad\u0db1\u0dd2\u0dba\u0dcf\u0dc0\u0dad\u0dca\u0d9a\u0dcf\u0dbd\u0dd3\u0db1 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0db8\u0dd4\u0dc5\u0dd4 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0d9c\u0dab\u0db1 </p>\n",
|
||||
"<p>train </p>\n": "<p>\u0daf\u0dd4\u0db8\u0dca\u0dbb\u0dd2\u0dba </p>\n",
|
||||
"<p>train the model </p>\n": "<p>\u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba\u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dbb\u0db1\u0dca\u0db1 </p>\n",
|
||||
"<p>\u2699\ufe0f Clip range. </p>\n": "<p>\u2699\ufe0f\u0d9a\u0dca\u0dbd\u0dd2\u0db4\u0dca \u0db4\u0dbb\u0dcf\u0dc3\u0dba. </p>\n",
|
||||
"<p>\u2699\ufe0f Entropy bonus coefficient. You can change this while the experiment is running. <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f/hyper_params\"><span translate=no>_^_0_^_</span></a> </p>\n": "<p>\u2699\ufe0f\u0d91\u0db1\u0dca\u0da7\u0dca\u0dbb\u0ddc\u0db4\u0dd2 \u0db4\u0dca\u0dbb\u0dc3\u0dcf\u0daf \u0dc3\u0d82\u0d9c\u0dd4\u0dab\u0d9a\u0dba. \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc0\u0db1 \u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0d94\u0db6\u0da7 \u0db8\u0dd9\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f/hyper_params\"><span translate=no>_^_0_^_</span></a> </p>\n",
|
||||
"<p>\u2699\ufe0f Number of epochs to train the model with sampled data. You can change this while the experiment is running. <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f/hyper_params\"><span translate=no>_^_0_^_</span></a> </p>\n": "<p>\u2699\ufe0f\u0db1\u0dd2\u0dba\u0dd0\u0daf\u0dd2 \u0daf\u0dad\u0dca\u0dad \u0dc3\u0db8\u0d9f \u0d86\u0d9a\u0dd8\u0dad\u0dd2\u0dba \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d91\u0db4\u0ddc\u0da0\u0dca \u0d9c\u0dab\u0db1. \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc0\u0db1 \u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0d94\u0db6\u0da7 \u0db8\u0dd9\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f/hyper_params\"><span translate=no>_^_0_^_</span></a> </p>\n",
|
||||
"<p>\u2699\ufe0f Value loss coefficient. You can change this while the experiment is running. <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f/hyper_params\"><span translate=no>_^_0_^_</span></a> </p>\n": "<p>\u2699\ufe0f\u0d85\u0d9c\u0dba \u0d85\u0dc4\u0dd2\u0db8\u0dd2 \u0dc3\u0d82\u0d9c\u0dd4\u0dab\u0d9a\u0dba. \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0dc0\u0db1 \u0d85\u0dad\u0dbb\u0dad\u0dd4\u0dbb \u0d94\u0db6\u0da7 \u0db8\u0dd9\u0dba \u0dc0\u0dd9\u0db1\u0dc3\u0dca \u0d9a\u0dc5 \u0dc4\u0dd0\u0d9a\u0dd2\u0dba. <a href=\"https://app.labml.ai/run/6eff28a0910e11eb9b008db315936e2f/hyper_params\"><span translate=no>_^_0_^_</span></a> </p>\n",
|
||||
"Annotated implementation to train a PPO agent on Atari Breakout game.": "Atari Breakout \u0d9a\u0dca\u0dbb\u0dd3\u0da9\u0dcf\u0dc0 \u0db4\u0dd2\u0dc5\u0dd2\u0db6\u0db3 PPO \u0db1\u0dd2\u0dba\u0ddd\u0da2\u0dd2\u0dad\u0dba\u0dd9\u0d9a\u0dd4 \u0db4\u0dd4\u0dc4\u0dd4\u0dab\u0dd4 \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8.",
|
||||
"PPO Experiment with Atari Breakout": "\u0d85\u0da7\u0dcf\u0dbb\u0dd2 \u0d9a\u0da9\u0dcf\u0dc0\u0dd0\u0da7\u0dd3\u0db8 \u0dc3\u0db8\u0d9f PPO \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8"
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"<h1>PPO Experiment with Atari Breakout</h1>\n<p>This experiment trains Proximal Policy Optimization (PPO) agent Atari Breakout game on OpenAI Gym. It runs the <a href=\"../game.html\">game environments on multiple processes</a> to sample efficiently.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n": "<h1>PPO \u4e0e Atari Breakout \u8fdb\u884c\u5b9e\u9a8c</h1>\n<p>\u8be5\u5b9e\u9a8c\u5728OpenAI Gym\u4e0a\u8bad\u7ec3\u4e86\u8fd1\u7aef\u7b56\u7565\u4f18\u5316\uff08PPO\uff09\u4ee3\u7406Atari Breakout\u6e38\u620f\u3002\u5b83\u5728<a href=\"../game.html\">\u591a\u4e2a\u8fdb\u7a0b\u4e0a\u8fd0\u884c\u6e38\u620f\u73af\u5883</a>\u4ee5\u9ad8\u6548\u91c7\u6837\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"<h2>Model</h2>\n": "<h2>\u578b\u53f7</h2>\n",
|
||||
"<h2>Run it</h2>\n": "<h2>\u8fd0\u884c\u5b83</h2>\n",
|
||||
"<h2>Trainer</h2>\n": "<h2>\u8bad\u7ec3\u5e08</h2>\n",
|
||||
"<h3>Calculate total loss</h3>\n": "<h3>\u8ba1\u7b97\u603b\u635f\u5931</h3>\n",
|
||||
"<h3>Destroy</h3>\n<p>Stop the workers</p>\n": "<h3>\u6467\u6bc1</h3>\n<p>\u963b\u6b62\u5de5\u4eba</p>\n",
|
||||
"<h3>Run training loop</h3>\n": "<h3>\u8dd1\u6b65\u8bad\u7ec3\u5faa\u73af</h3>\n",
|
||||
"<h3>Sample data with current policy</h3>\n": "<h3>\u5f53\u524d\u653f\u7b56\u7684\u6837\u672c\u6570\u636e</h3>\n",
|
||||
"<h3>Train the model based on samples</h3>\n": "<h3>\u6839\u636e\u6837\u672c\u8bad\u7ec3\u6a21\u578b</h3>\n",
|
||||
"<h4>Configurations</h4>\n": "<h4>\u914d\u7f6e</h4>\n",
|
||||
"<h4>Initialize</h4>\n": "<h4>\u521d\u59cb\u5316</h4>\n",
|
||||
"<h4>Normalize advantage function</h4>\n": "<h4>\u89c4\u8303\u5316\u4f18\u52bf\u51fd\u6570</h4>\n",
|
||||
"<p> </p>\n": "<p></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> keeps track of the last observation from each worker, which is the input for the model to sample the next action </p>\n": "<p><span translate=no>_^_0_^_</span>\u8ddf\u8e2a\u6765\u81ea\u6bcf\u4e2a worker \u7684\u6700\u540e\u4e00\u4e2a\u89c2\u6d4b\u503c\uff0c\u8fd9\u662f\u6a21\u578b\u5bf9\u4e0b\u4e00\u4e2a\u64cd\u4f5c\u8fdb\u884c\u91c7\u6837\u7684\u8f93\u5165</p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> returns sampled from <span translate=no>_^_1_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\u4ece\u4e2d\u62bd\u6837\u7684\u8fd4\u56de<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>, <span translate=no>_^_1_^_</span> are actions sampled from <span translate=no>_^_2_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span>\uff0c<span translate=no>_^_1_^_</span>\u662f\u4ece\u4e2d\u91c7\u6837\u7684\u52a8\u4f5c<span translate=no>_^_2_^_</span></p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span>, where <span translate=no>_^_1_^_</span> is advantages sampled from <span translate=no>_^_2_^_</span>. Refer to sampling function in <a href=\"#main\">Main class</a> below for the calculation of <span translate=no>_^_3_^_</span>. </p>\n": "<p><span translate=no>_^_0_^_</span>\uff0c\u4f18<span translate=no>_^_1_^_</span>\u52bf\u4ece\u54ea\u91cc\u62bd\u6837<span translate=no>_^_2_^_</span>\u3002\u6709\u5173\u8ba1\u7b97\uff0c\u8bf7\u53c2\u9605\u4ee5\u4e0b <a href=\"#main\">Main \u7c7b</a>\u4e2d\u7684\u91c7\u6837\u51fd\u6570<span translate=no>_^_3_^_</span>\u3002</p>\n",
|
||||
"<p>A fully connected layer takes the flattened frame from third convolution layer, and outputs 512 features </p>\n": "<p>\u5b8c\u5168\u8fde\u63a5\u7684\u56fe\u5c42\u4ece\u7b2c\u4e09\u4e2a\u5377\u79ef\u56fe\u5c42\u83b7\u53d6\u5e73\u5766\u7684\u5e27\uff0c\u5e76\u8f93\u51fa 512 \u4e2a\u8981\u7d20</p>\n",
|
||||
"<p>A fully connected layer to get logits for <span translate=no>_^_0_^_</span> </p>\n": "<p>\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42\uff0c\u7528\u4e8e\u83b7\u53d6\u65e5\u5fd7<span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>A fully connected layer to get value function </p>\n": "<p>\u4e00\u4e2a\u5b8c\u5168\u8fde\u63a5\u7684\u5c42\u6765\u83b7\u53d6\u4ef7\u503c\u51fd\u6570</p>\n",
|
||||
"<p>Add a new line to the screen periodically </p>\n": "<p>\u5b9a\u671f\u5728\u5c4f\u5e55\u4e0a\u6dfb\u52a0\u65b0\u884c</p>\n",
|
||||
"<p>Add to tracker </p>\n": "<p>\u6dfb\u52a0\u5230\u8ffd\u8e2a\u5668</p>\n",
|
||||
"<p>Calculate Entropy Bonus</p>\n<p><span translate=no>_^_0_^_</span> </p>\n": "<p>\u8ba1\u7b97\u71b5\u52a0\u6210</p>\n<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>Calculate gradients </p>\n": "<p>\u8ba1\u7b97\u68af\u5ea6</p>\n",
|
||||
"<p>Calculate policy loss </p>\n": "<p>\u8ba1\u7b97\u4fdd\u5355\u635f\u5931</p>\n",
|
||||
"<p>Calculate value function loss </p>\n": "<p>\u8ba1\u7b97\u503c\u51fd\u6570\u635f\u5931</p>\n",
|
||||
"<p>Clip gradients </p>\n": "<p>\u526a\u8f91\u6e10\u53d8</p>\n",
|
||||
"<p>Clipping range </p>\n": "<p>\u88c1\u526a\u8303\u56f4</p>\n",
|
||||
"<p>Configurations </p>\n": "<p>\u914d\u7f6e</p>\n",
|
||||
"<p>Create the experiment </p>\n": "<p>\u521b\u5efa\u5b9e\u9a8c</p>\n",
|
||||
"<p>Entropy bonus coefficient </p>\n": "<p>\u71b5\u52a0\u6210\u7cfb\u6570</p>\n",
|
||||
"<p>GAE with <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span> </p>\n": "<p>\u4f7f\u7528<span translate=no>_^_0_^_</span>\u548c\u7684 GAE<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Get value of after the final step </p>\n": "<p>\u5728\u6700\u540e\u4e00\u6b65\u4e4b\u540e\u83b7\u53d6\u7684\u503c</p>\n",
|
||||
"<p>Initialize the trainer </p>\n": "<p>\u521d\u59cb\u5316\u8bad\u7ec3\u5668</p>\n",
|
||||
"<p>It learns faster with a higher number of epochs, but becomes a little unstable; that is, the average episode reward does not monotonically increase over time. May be reducing the clipping range might solve it. </p>\n": "<p>\u968f\u7740\u65f6\u4ee3\u6570\u91cf\u7684\u589e\u52a0\uff0c\u5b83\u5b66\u4e60\u5f97\u66f4\u5feb\uff0c\u4f46\u4f1a\u53d8\u5f97\u6709\u70b9\u4e0d\u7a33\u5b9a\uff1b\u4e5f\u5c31\u662f\u8bf4\uff0c\u5e73\u5747\u5267\u96c6\u5956\u52b1\u4e0d\u4f1a\u968f\u7740\u65f6\u95f4\u7684\u63a8\u79fb\u800c\u5355\u8c03\u589e\u52a0\u3002\u53ef\u80fd\u4f1a\u7f29\u5c0f\u526a\u5207\u8303\u56f4\u53ef\u80fd\u4f1a\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\u3002</p>\n",
|
||||
"<p>Learning rate </p>\n": "<p>\u5b66\u4e60\u7387</p>\n",
|
||||
"<p>Number of mini batches </p>\n": "<p>\u5fae\u578b\u6279\u6b21\u6570</p>\n",
|
||||
"<p>Number of steps to run on each process for a single update </p>\n": "<p>\u5355\u6b21\u66f4\u65b0\u7684\u6bcf\u4e2a\u8fdb\u7a0b\u8981\u8fd0\u884c\u7684\u6b65\u9aa4\u6570</p>\n",
|
||||
"<p>Number of updates </p>\n": "<p>\u66f4\u65b0\u6b21\u6570</p>\n",
|
||||
"<p>Number of worker processes </p>\n": "<p>\u5de5\u4f5c\u8fdb\u7a0b\u6570</p>\n",
|
||||
"<p>PPO Loss </p>\n": "<p>PPO \u635f\u5931</p>\n",
|
||||
"<p>Run and monitor the experiment </p>\n": "<p>\u8fd0\u884c\u5e76\u76d1\u63a7\u5b9e\u9a8c</p>\n",
|
||||
"<p>Sampled observations are fed into the model to get <span translate=no>_^_0_^_</span> and <span translate=no>_^_1_^_</span>; we are treating observations as state </p>\n": "<p>\u91c7\u6837\u89c2\u6d4b\u503c\u88ab\u8f93\u5165\u5230\u6a21\u578b\u4e2d\u4ee5\u83b7\u53d6<span translate=no>_^_0_^_</span>\u548c<span translate=no>_^_1_^_</span>\uff1b\u6211\u4eec\u5c06\u89c2\u6d4b\u503c\u89c6\u4e3a\u72b6\u6001</p>\n",
|
||||
"<p>Save tracked indicators. </p>\n": "<p>\u4fdd\u5b58\u8ddf\u8e2a\u7684\u6307\u6807\u3002</p>\n",
|
||||
"<p>Scale observations from <span translate=no>_^_0_^_</span> to <span translate=no>_^_1_^_</span> </p>\n": "<p>\u5c06\u89c2\u6d4b\u503c\u4ece\u7f29\u653e<span translate=no>_^_0_^_</span>\u5230<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>Select device </p>\n": "<p>\u9009\u62e9\u8bbe\u5907</p>\n",
|
||||
"<p>Set learning rate </p>\n": "<p>\u8bbe\u7f6e\u5b66\u4e60\u901f\u7387</p>\n",
|
||||
"<p>Stop the workers </p>\n": "<p>\u963b\u6b62\u5de5\u4eba</p>\n",
|
||||
"<p>The first convolution layer takes a 84x84 frame and produces a 20x20 frame </p>\n": "<p>\u7b2c\u4e00\u4e2a\u5377\u79ef\u5c42\u91c7\u7528 84x84 \u5e27\u5e76\u751f\u6210 20x20 \u5e27</p>\n",
|
||||
"<p>The second convolution layer takes a 20x20 frame and produces a 9x9 frame </p>\n": "<p>\u7b2c\u4e8c\u4e2a\u5377\u79ef\u5c42\u91c7\u7528 20x20 \u5e27\u5e76\u751f\u6210 9x9 \u7684\u5e27</p>\n",
|
||||
"<p>The third convolution layer takes a 9x9 frame and produces a 7x7 frame </p>\n": "<p>\u7b2c\u4e09\u4e2a\u5377\u79ef\u5c42\u91c7\u7528 9x9 \u5e27\u5e76\u751f\u6210 7x7 \u5e27</p>\n",
|
||||
"<p>Update parameters based on gradients </p>\n": "<p>\u6839\u636e\u6e10\u53d8\u66f4\u65b0\u53c2\u6570</p>\n",
|
||||
"<p>Value Loss </p>\n": "<p>\u4ef7\u503c\u635f\u5931</p>\n",
|
||||
"<p>Value loss coefficient </p>\n": "<p>\u4ef7\u503c\u635f\u5931\u7cfb\u6570</p>\n",
|
||||
"<p>You can change this while the experiment is running. \u2699\ufe0f Learning rate. </p>\n": "<p>\u4f60\u53ef\u4ee5\u5728\u5b9e\u9a8c\u8fd0\u884c\u65f6\u66f4\u6539\u6b64\u8bbe\u7f6e\u3002\u2699\ufe0f \u5b66\u4e60\u7387\u3002</p>\n",
|
||||
"<p>Zero out the previously calculated gradients </p>\n": "<p>\u5c06\u5148\u524d\u8ba1\u7b97\u7684\u68af\u5ea6\u5f52\u96f6</p>\n",
|
||||
"<p>calculate advantages </p>\n": "<p>\u8ba1\u7b97\u4f18\u52bf</p>\n",
|
||||
"<p>collect episode info, which is available if an episode finished; this includes total reward and length of the episode - look at <span translate=no>_^_0_^_</span> to see how it works. </p>\n": "<p>\u6536\u96c6\u5267\u96c6\u4fe1\u606f\uff0c\u5728\u5267\u96c6\u7ed3\u675f\u540e\u53ef\u7528\uff1b\u8fd9\u5305\u62ec\u603b\u5956\u52b1\u548c\u5267\u96c6\u957f\u5ea6\u2014\u2014\u770b\u770b<span translate=no>_^_0_^_</span>\u5b83\u662f\u5982\u4f55\u8fd0\u4f5c\u7684\u3002</p>\n",
|
||||
"<p>create workers </p>\n": "<p>\u521b\u5efa\u5de5\u4f5c\u4eba\u5458</p>\n",
|
||||
"<p>for each mini batch </p>\n": "<p>\u6bcf\u5c0f\u6279\u6b21</p>\n",
|
||||
"<p>for monitoring </p>\n": "<p>\u7528\u4e8e\u76d1\u63a7</p>\n",
|
||||
"<p>get mini batch </p>\n": "<p>\u83b7\u5f97\u5c0f\u6279\u91cf</p>\n",
|
||||
"<p>get results after executing the actions </p>\n": "<p>\u6267\u884c\u64cd\u4f5c\u540e\u83b7\u5f97\u7ed3\u679c</p>\n",
|
||||
"<p>initialize tensors for observations </p>\n": "<p>\u521d\u59cb\u5316\u89c2\u6d4b\u503c\u7684\u5f20\u91cf</p>\n",
|
||||
"<p>last 100 episode information </p>\n": "<p>\u6700\u8fd1 100 \u96c6\u4fe1\u606f</p>\n",
|
||||
"<p>model </p>\n": "<p>\u6a21\u578b</p>\n",
|
||||
"<p>number of epochs to train the model with sampled data </p>\n": "<p>\u4f7f\u7528\u91c7\u6837\u6570\u636e\u8bad\u7ec3\u6a21\u578b\u7684\u5468\u671f\u6570</p>\n",
|
||||
"<p>number of mini batches </p>\n": "<p>\u5fae\u578b\u6279\u6b21\u6570</p>\n",
|
||||
"<p>number of steps to run on each process for a single update </p>\n": "<p>\u5355\u6b21\u66f4\u65b0\u7684\u6bcf\u4e2a\u8fdb\u7a0b\u8981\u8fd0\u884c\u7684\u6b65\u9aa4\u6570</p>\n",
|
||||
"<p>number of updates </p>\n": "<p>\u66f4\u65b0\u6b21\u6570</p>\n",
|
||||
"<p>number of worker processes </p>\n": "<p>\u5de5\u4f5c\u8fdb\u7a0b\u7684\u6570\u91cf</p>\n",
|
||||
"<p>optimizer </p>\n": "<p>\u4f18\u5316\u8005</p>\n",
|
||||
"<p>run sampled actions on each worker </p>\n": "<p>\u5bf9\u6bcf\u4e2a worker \u8fd0\u884c\u91c7\u6837\u64cd\u4f5c</p>\n",
|
||||
"<p>sample <span translate=no>_^_0_^_</span> from each worker </p>\n": "<p>\u6bcf\u4f4d\u5de5\u4f5c\u4eba\u5458<span translate=no>_^_0_^_</span>\u7684\u6837\u672c</p>\n",
|
||||
"<p>sample actions from <span translate=no>_^_0_^_</span> for each worker; this returns arrays of size <span translate=no>_^_1_^_</span> </p>\n": "<p>\u6bcf\u4e2a worker<span translate=no>_^_0_^_</span> \u7684\u793a\u4f8b\u64cd\u4f5c\uff1b\u8fd9\u4f1a\u8fd4\u56de\u5927\u5c0f\u6570\u7ec4<span translate=no>_^_1_^_</span></p>\n",
|
||||
"<p>sample with current policy </p>\n": "<p>\u5f53\u524d\u653f\u7b56\u7684\u6837\u672c</p>\n",
|
||||
"<p>samples are currently in <span translate=no>_^_0_^_</span> table, we should flatten it for training </p>\n": "<p>\u6837\u672c\u76ee\u524d\u5728<span translate=no>_^_0_^_</span>\u8868\u4e2d\uff0c\u6211\u4eec\u5e94\u8be5\u5c06\u5176\u538b\u5e73\u4ee5\u8fdb\u884c\u8bad\u7ec3</p>\n",
|
||||
"<p>shuffle for each epoch </p>\n": "<p>\u968f\u673a\u64ad\u653e\u6bcf\u4e2a\u65f6\u4ee3</p>\n",
|
||||
"<p>size of a mini batch </p>\n": "<p>\u5c0f\u6279\u91cf\u7684\u5927\u5c0f</p>\n",
|
||||
"<p>total number of samples for a single update </p>\n": "<p>\u5355\u6b21\u66f4\u65b0\u7684\u6837\u672c\u603b\u6570</p>\n",
|
||||
"<p>train </p>\n": "<p>\u706b\u8f66</p>\n",
|
||||
"<p>train the model </p>\n": "<p>\u8bad\u7ec3\u6a21\u578b</p>\n",
|
||||
"<p>\u2699\ufe0f Clip range. </p>\n": "<p>\u2699\ufe0f \u526a\u8f91\u8303\u56f4\u3002</p>\n",
|
||||
"<p>\u2699\ufe0f Entropy bonus coefficient. You can change this while the experiment is running. </p>\n": "<p>\u2699\ufe0f \u71b5\u52a0\u6210\u7cfb\u6570\u3002\u4f60\u53ef\u4ee5\u5728\u5b9e\u9a8c\u8fd0\u884c\u65f6\u66f4\u6539\u6b64\u8bbe\u7f6e\u3002</p>\n",
|
||||
"<p>\u2699\ufe0f Number of epochs to train the model with sampled data. You can change this while the experiment is running. </p>\n": "<p>\u2699\ufe0f \u4f7f\u7528\u91c7\u6837\u6570\u636e\u8bad\u7ec3\u6a21\u578b\u7684\u65f6\u4ee3\u6570\u3002\u4f60\u53ef\u4ee5\u5728\u5b9e\u9a8c\u8fd0\u884c\u65f6\u66f4\u6539\u6b64\u8bbe\u7f6e\u3002</p>\n",
|
||||
"<p>\u2699\ufe0f Value loss coefficient. You can change this while the experiment is running. </p>\n": "<p>\u2699\ufe0f \u4ef7\u503c\u635f\u5931\u7cfb\u6570\u3002\u4f60\u53ef\u4ee5\u5728\u5b9e\u9a8c\u8fd0\u884c\u65f6\u66f4\u6539\u6b64\u8bbe\u7f6e\u3002</p>\n",
|
||||
"Annotated implementation to train a PPO agent on Atari Breakout game.": "\u5e26\u6ce8\u91ca\u7684\u5b9e\u73b0\uff0c\u7528\u4e8e\u5728 Atari Breakout \u6e38\u620f\u4e2d\u8bad\u7ec3 PPO \u7279\u5de5\u3002",
|
||||
"PPO Experiment with Atari Breakout": "PPO \u4f7f\u7528 Atari Breakout \u8fdb\u884c\u5b9e\u9a8c"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"<h1>Generalized Advantage Estimation (GAE)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of paper <a href=\"https://arxiv.org/abs/1506.02438\">Generalized Advantage Estimation</a>.</p>\n<p>You can find an experiment that uses it <a href=\"experiment.html\">here</a>.</p>\n": "<h1>\u4e00\u822c\u5316\u512a\u4f4d\u6027\u63a8\u5b9a (GAE)</h1>\n<p><a href=\"https://pytorch.org\"><a href=\"https://arxiv.org/abs/1506.02438\">\u3053\u308c\u306f\u7d19\u306e\u4e00\u822c\u5316\u30a2\u30c9\u30d0\u30f3\u30c6\u30fc\u30b8\u63a8\u5b9a\u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059</a></a>\u3002</p>\n<p><a href=\"experiment.html\">\u3053\u308c\u3092\u4f7f\u3063\u305f\u5b9f\u9a13\u306f\u3053\u3061\u3089\u304b\u3089\u3054\u89a7\u3044\u305f\u3060\u3051\u307e\u3059</a>\u3002</p>\n",
|
||||
"<h3>Calculate advantages</h3>\n<span translate=no>_^_0_^_</span><p><span translate=no>_^_1_^_</span> is high bias, low variance, whilst <span translate=no>_^_2_^_</span> is unbiased, high variance.</p>\n<p>We take a weighted average of <span translate=no>_^_3_^_</span> to balance bias and variance. This is called Generalized Advantage Estimation. <span translate=no>_^_4_^_</span> We set <span translate=no>_^_5_^_</span>, this gives clean calculation for <span translate=no>_^_6_^_</span></p>\n<span translate=no>_^_7_^_</span>": "<h3>\u5229\u70b9\u3092\u8a08\u7b97</h3>\n<span translate=no>_^_0_^_</span><p><span translate=no>_^_1_^_</span>\u30d0\u30a4\u30a2\u30b9\u304c\u9ad8\u304f\u5206\u6563\u304c\u5c0f\u3055\u304f\u3001\u504f\u308a\u304c\u306a\u304f\u3001<span translate=no>_^_2_^_</span>\u5206\u6563\u304c\u5927\u304d\u3044\u3002</p>\n<p><span translate=no>_^_3_^_</span>\u30d0\u30a4\u30a2\u30b9\u3068\u5206\u6563\u306e\u30d0\u30e9\u30f3\u30b9\u3092\u53d6\u308b\u305f\u3081\u306b\u3001\u52a0\u91cd\u5e73\u5747\u3092\u53d6\u308a\u307e\u3059\u3002\u3053\u308c\u306f\u4e00\u822c\u5316\u30a2\u30c9\u30d0\u30f3\u30c6\u30fc\u30b8\u63a8\u5b9a\u3068\u547c\u3070\u308c\u307e\u3059\u3002<span translate=no>_^_4_^_</span>\u8a2d\u5b9a\u3057\u307e\u3057\u305f\u3002\u3053\u308c\u306b\u3088\u308a<span translate=no>_^_5_^_</span>\u3001\u8a08\u7b97\u304c\u304d\u308c\u3044\u306b\u306a\u308a\u307e\u3059 <span translate=no>_^_6_^_</span></p>\n<span translate=no>_^_7_^_</span>",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>advantages table </p>\n": "<p>\u5229\u70b9\u8868</p>\n",
|
||||
"<p>mask if episode completed after step <span translate=no>_^_0_^_</span> </p>\n": "<p>\u30b9\u30c6\u30c3\u30d7\u306e\u5f8c\u306b\u30a8\u30d4\u30bd\u30fc\u30c9\u304c\u5b8c\u4e86\u3057\u305f\u5834\u5408\u306f\u30de\u30b9\u30af <span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>note that we are collecting in reverse order. <em>My initial code was appending to a list and I forgot to reverse it later. It took me around 4 to 5 hours to find the bug. The performance of the model was improving slightly during initial runs, probably because the samples are similar.</em> </p>\n": "<p>\u9006\u306e\u9806\u5e8f\u3067\u53ce\u96c6\u3057\u3066\u3044\u308b\u3053\u3068\u306b\u6ce8\u610f\u3057\u3066\u304f\u3060\u3055\u3044\u3002<em>\u6700\u521d\u306e\u30b3\u30fc\u30c9\u306f\u30ea\u30b9\u30c8\u306b\u8ffd\u52a0\u3055\u308c\u3066\u3044\u3066\u3001\u5f8c\u3067\u5143\u306b\u623b\u3059\u306e\u3092\u5fd8\u308c\u307e\u3057\u305f\u3002\u30d0\u30b0\u3092\u898b\u3064\u3051\u308b\u306e\u306b\u7d044\u301c5\u6642\u9593\u304b\u304b\u308a\u307e\u3057\u305f\u3002\u30e2\u30c7\u30eb\u306e\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u306f\u3001\u304a\u305d\u3089\u304f\u30b5\u30f3\u30d7\u30eb\u304c\u4f3c\u3066\u3044\u308b\u305f\u3081\u304b\u3001\u6700\u521d\u306e\u5b9f\u884c\u6642\u306b\u308f\u305a\u304b\u306b\u5411\u4e0a\u3057\u3066\u3044\u307e\u3057\u305f\u3002</em></p>\n",
|
||||
"A PyTorch implementation/tutorial of Generalized Advantage Estimation (GAE).": "\u4e00\u822c\u5316\u30a2\u30c9\u30d0\u30f3\u30c6\u30fc\u30b8\u63a8\u5b9a (GAE) \u306e PyTorch \u5b9f\u88c5/\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb\u3002",
|
||||
"Generalized Advantage Estimation (GAE)": "\u4e00\u822c\u5316\u512a\u4f4d\u6027\u63a8\u5b9a (GAE)"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"<h1>Generalized Advantage Estimation (GAE)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of paper <a href=\"https://arxiv.org/abs/1506.02438\">Generalized Advantage Estimation</a>.</p>\n<p>You can find an experiment that uses it <a href=\"experiment.html\">here</a>.</p>\n": "<h1>\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba\u0d9a\u0dc5 \u0dc0\u0dcf\u0dc3\u0dd2 \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0dc0 (GAE)</h1>\n<p>\u0db8\u0dd9\u0dba <a href=\"https://pytorch.org\">PyTorch</a> \u0d9a\u0da9\u0daf\u0dcf\u0dc3\u0dd2 <a href=\"https://arxiv.org/abs/1506.02438\">\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dbb\u0db1 \u0dbd\u0daf \u0dc0\u0dcf\u0dc3\u0dd2 \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4 \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dd2</a> . </p>\n<p>\u0d91\u0dba\u0db7\u0dcf\u0dc0\u0dd2\u0dad\u0dcf \u0d9a\u0dbb\u0db1 \u0d85\u0dad\u0dca\u0dc4\u0daf\u0dcf \u0db6\u0dd0\u0dbd\u0dd3\u0db8\u0d9a\u0dca \u0d94\u0db6\u0da7 \u0dc3\u0ddc\u0dba\u0dcf\u0d9c\u0dad \u0dc4\u0dd0\u0d9a\u0dd2\u0dba <a href=\"experiment.html\">\u0db8\u0dd9\u0dc4\u0dd2</a>. </p>\n",
|
||||
"<h3>Calculate advantages</h3>\n<span translate=no>_^_0_^_</span><p><span translate=no>_^_1_^_</span> is high bias, low variance, whilst <span translate=no>_^_2_^_</span> is unbiased, high variance.</p>\n<p>We take a weighted average of <span translate=no>_^_3_^_</span> to balance bias and variance. This is called Generalized Advantage Estimation. <span translate=no>_^_4_^_</span> We set <span translate=no>_^_5_^_</span>, this gives clean calculation for <span translate=no>_^_6_^_</span></p>\n<span translate=no>_^_7_^_</span>": "<h3>\u0dc0\u0dcf\u0dc3\u0dd2\u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dbb\u0db1\u0dca\u0db1</h3>\n<span translate=no>_^_0_^_</span><p><span translate=no>_^_1_^_</span> \u0d89\u0dc4\u0dc5 \u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0, \u0d85\u0da9\u0dd4 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0, \u0d85\u0db4\u0d9a\u0dca\u0dc2\u0db4\u0dcf\u0dad\u0dd3 <span translate=no>_^_2_^_</span> \u0dc0\u0db1 \u0d85\u0dad\u0dbb \u0d89\u0dc4\u0dc5 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0. </p>\n<p>\u0db1\u0dd0\u0db9\u0dd4\u0dbb\u0dd4\u0dc0\u0dc3\u0dc4 \u0dc0\u0dd2\u0da0\u0dbd\u0dad\u0dcf\u0dc0 \u0dc3\u0db8\u0dad\u0dd4\u0dbd\u0dd2\u0dad <span translate=no>_^_3_^_</span> \u0d9a\u0dd2\u0dbb\u0dd3\u0db8 \u0dc3\u0db3\u0dc4\u0dcf \u0d85\u0db4\u0dd2 \u0db6\u0dbb \u0dad\u0dd0\u0db6\u0dd6 \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0dba\u0d9a\u0dca \u0d9c\u0db1\u0dd2\u0db8\u0dd4. \u0db8\u0dd9\u0dba \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dc5 \u0dc0\u0dcf\u0dc3\u0dd2 \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0dc0 \u0dbd\u0dd9\u0dc3 \u0dc4\u0dd0\u0db3\u0dd2\u0db1\u0dca\u0dc0\u0dda. <span translate=no>_^_4_^_</span> \u0d85\u0db4\u0dd2 \u0dc3\u0d9a\u0dc3\u0dca \u0d9a\u0dc5\u0dd9\u0db8\u0dd4 <span translate=no>_^_5_^_</span>, \u0db8\u0dd9\u0dba \u0db4\u0dd2\u0dbb\u0dd2\u0dc3\u0dd2\u0daf\u0dd4 \u0d9c\u0dab\u0db1\u0dba \u0d9a\u0dd2\u0dbb\u0dd3\u0db8\u0d9a\u0dca \u0dbd\u0db6\u0dcf \u0daf\u0dd9\u0dba\u0dd2 <span translate=no>_^_6_^_</span></p>\n<span translate=no>_^_7_^_</span>",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>advantages table </p>\n": "<p>\u0dc0\u0dcf\u0dc3\u0dd2\u0dc0\u0d9c\u0dd4\u0dc0 </p>\n",
|
||||
"<p>mask if episode completed after step <span translate=no>_^_0_^_</span> </p>\n": "<p>\u0db4\u0dd2\u0dba\u0dc0\u0dbb\u0dd9\u0db1\u0dca\u0db4\u0dc3\u0dd4 \u0d9a\u0dae\u0dcf\u0d82\u0d9c\u0dba \u0dc3\u0db8\u0dca\u0db4\u0dd6\u0dbb\u0dca\u0dab \u0dc0\u0dd4\u0dc0\u0dc4\u0ddc\u0dad\u0dca \u0dc0\u0dd9\u0dc3\u0dca \u0db8\u0dd4\u0dc4\u0dd4\u0dab <span translate=no>_^_0_^_</span> </p>\n",
|
||||
"<p>note that we are collecting in reverse order. <em>My initial code was appending to a list and I forgot to reverse it later. It took me around 4 to 5 hours to find the bug. The performance of the model was improving slightly during initial runs, probably because the samples are similar.</em> </p>\n": "<p>\u0d85\u0db4\u0dd2\u0db4\u0dca\u0dbb\u0dad\u0dd2\u0dbd\u0ddd\u0db8 \u0d85\u0db1\u0dd4\u0db4\u0dd2\u0dc5\u0dd2\u0dc0\u0dd9\u0dbd\u0dd2\u0db1\u0dca \u0d91\u0d9a\u0dad\u0dd4 \u0d9a\u0dbb\u0db1 \u0db6\u0dc0 \u0dc3\u0dbd\u0d9a\u0db1\u0dca\u0db1. <em>\u0db8\u0d9c\u0dda\u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a \u0d9a\u0dda\u0dad\u0dba \u0dbd\u0dd0\u0dba\u0dd2\u0dc3\u0dca\u0dad\u0dd4\u0dc0\u0d9a\u0da7 \u0d91\u0d9a\u0dad\u0dd4 \u0dc0\u0dd9\u0db8\u0dd2\u0db1\u0dca \u0dad\u0dd2\u0db6\u0dd6 \u0d85\u0dad\u0dbb \u0db4\u0dc3\u0dd4\u0dc0 \u0d91\u0dba \u0d86\u0db4\u0dc3\u0dd4 \u0dc4\u0dd0\u0dbb\u0dc0\u0dd3\u0db8\u0da7 \u0db8\u0da7 \u0d85\u0db8\u0dad\u0d9a \u0dc0\u0dd2\u0dba. \u0daf\u0ddd\u0dc2\u0dba \u0dc3\u0ddc\u0dba\u0dcf \u0d9c\u0dd0\u0db1\u0dd3\u0db8\u0da7 \u0db8\u0da7 \u0db4\u0dd0\u0dba 4 \u0dc3\u0dd2\u0da7 5 \u0daf\u0d9a\u0dca\u0dc0\u0dcf \u0d9c\u0dad \u0dc0\u0dd2\u0dba. \u0d86\u0daf\u0dbb\u0dca\u0dc1 \u0d9a\u0dcf\u0dbb\u0dca\u0dba \u0dc3\u0dcf\u0db0\u0db1\u0dba \u0d86\u0dbb\u0db8\u0dca\u0db7\u0d9a \u0dbd\u0d9a\u0dd4\u0dab\u0dd4 \u0dad\u0dd4\u0dc5 \u0dad\u0dbb\u0db8\u0d9a\u0dca \u0dc0\u0dd0\u0da9\u0dd2 \u0daf\u0dd2\u0dba\u0dd4\u0dab\u0dd4 \u0dc0\u0dd2\u0dba, \u0db6\u0ddc\u0dc4\u0ddd \u0dc0\u0dd2\u0da7 \u0dc3\u0dcf\u0db8\u0dca\u0db4\u0dbd \u0dc3\u0db8\u0dcf\u0db1 \u0db1\u0dd2\u0dc3\u0dcf. </em> </p>\n",
|
||||
"A PyTorch implementation/tutorial of Generalized Advantage Estimation (GAE).": "PyTorch \u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba \u0dc0\u0dcf\u0dc3\u0dd2 \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4 (GAE) \u0dc4\u0dd2 PyTorch \u0d9a\u0dca\u0dbb\u0dd2\u0dba\u0dcf\u0dad\u0dca\u0db8\u0d9a \u0d9a\u0dd2\u0dbb\u0dd3\u0db8/\u0db1\u0dd2\u0db6\u0db1\u0dca\u0db0\u0db1\u0dba.",
|
||||
"Generalized Advantage Estimation (GAE)": "\u0dc3\u0dcf\u0db8\u0dcf\u0db1\u0dca\u0dba\u0d9a\u0dbb\u0dab\u0dba \u0d9a\u0dc5 \u0dc0\u0dcf\u0dc3\u0dd2 \u0d87\u0dc3\u0dca\u0dad\u0db8\u0dda\u0db1\u0dca\u0dad\u0dd4\u0dc0 (GAE)"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"<h1>Generalized Advantage Estimation (GAE)</h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of paper <a href=\"https://arxiv.org/abs/1506.02438\">Generalized Advantage Estimation</a>.</p>\n<p>You can find an experiment that uses it <a href=\"experiment.html\">here</a>.</p>\n": "<h1>\u5e7f\u4e49\u4f18\u52bf\u4f30\u8ba1 (GAE)</h1>\n<p>\u8fd9\u662f\u8bba\u6587<a href=\"https://arxiv.org/abs/1506.02438\">\u5e7f\u4e49\u4f18\u52bf\u4f30\u8ba1</a>\u7684 <a href=\"https://pytorch.org\">PyTorch</a> \u5b9e\u73b0\u3002</p>\n<p>\u4f60\u53ef\u4ee5<a href=\"experiment.html\">\u5728\u8fd9\u91cc</a>\u627e\u5230\u4e00\u4e2a\u4f7f\u7528\u5b83\u7684\u5b9e\u9a8c\u3002</p>\n",
|
||||
"<h3>Calculate advantages</h3>\n<span translate=no>_^_0_^_</span><p><span translate=no>_^_1_^_</span> is high bias, low variance, whilst <span translate=no>_^_2_^_</span> is unbiased, high variance.</p>\n<p>We take a weighted average of <span translate=no>_^_3_^_</span> to balance bias and variance. This is called Generalized Advantage Estimation. <span translate=no>_^_4_^_</span> We set <span translate=no>_^_5_^_</span>, this gives clean calculation for <span translate=no>_^_6_^_</span></p>\n<span translate=no>_^_7_^_</span>": "<h3>\u8ba1\u7b97\u4f18\u52bf</h3>\n<span translate=no>_^_0_^_</span><p><span translate=no>_^_1_^_</span>\u662f\u9ad8\u504f\u5dee\uff0c\u4f4e\u65b9\u5dee\uff0c\u800c<span translate=no>_^_2_^_</span>\u65e0\u504f\u5dee\uff0c\u9ad8\u65b9\u5dee\u3002</p>\n<p>\u6211\u4eec\u91c7\u7528\u52a0\u6743\u5e73\u5747\u503c<span translate=no>_^_3_^_</span>\u6765\u5e73\u8861\u504f\u5dee\u548c\u65b9\u5dee\u3002\u8fd9\u79f0\u4e3a\u5e7f\u4e49\u4f18\u52bf\u4f30\u8ba1\u3002<span translate=no>_^_4_^_</span>\u6211\u4eec\u8bbe\u7f6e<span translate=no>_^_5_^_</span>\uff0c\u8fd9\u7ed9\u51fa\u4e86\u5e72\u51c0\u7684\u8ba1\u7b97<span translate=no>_^_6_^_</span></p>\n<span translate=no>_^_7_^_</span>",
|
||||
"<p> </p>\n": "<p> </p>\n",
|
||||
"<p><span translate=no>_^_0_^_</span> </p>\n": "<p><span translate=no>_^_0_^_</span></p>\n",
|
||||
"<p>advantages table </p>\n": "<p>\u4f18\u52bf\u8868</p>\n",
|
||||
"<p>mask if episode completed after step <span translate=no>_^_0_^_</span> </p>\n": "<p>\u5982\u679c\u5267\u96c6\u5728\u6b65\u9aa4\u4e4b\u540e\u5b8c\u6210\uff0c\u8bf7\u63a9\u76d6<span translate=no>_^_0_^_</span></p>\n",
|
||||
"A PyTorch implementation/tutorial of Generalized Advantage Estimation (GAE).": "\u5e7f\u4e49\u4f18\u52bf\u4f30\u8ba1\uff08GAE\uff09\u7684 PyTorch \u5b9e\u73b0/\u6559\u7a0b\u3002",
|
||||
"Generalized Advantage Estimation (GAE)": "\u5e7f\u4e49\u4f18\u52bf\u4f30\u8ba1 (GAE)"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/rl/ppo/index.html\">Proximal Policy Optimization - PPO</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1707.06347\">Proximal Policy Optimization - PPO</a>.</p>\n<p>PPO is a policy gradient method for reinforcement learning. Simple policy gradient methods one do a single gradient update per sample (or a set of samples). Doing multiple gradient steps for a singe sample causes problems because the policy deviates too much producing a bad policy. PPO lets us do multiple gradient updates per sample by trying to keep the policy close to the policy that was used to sample data. It does so by clipping gradient flow if the updated policy is not close to the policy used to sample the data.</p>\n<p>You can find an experiment that uses it <a href=\"https://nn.labml.ai/rl/ppo/experiment.html\">here</a>. The experiment uses <a href=\"https://nn.labml.ai/rl/ppo/gae.html\">Generalized Advantage Estimation</a>.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/rl/ppo/index.html\">\u8fd1\u63a5\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316-PPO</a></h1>\n<p><a href=\"https://arxiv.org/abs/1707.06347\">\u3053\u308c\u306f\u8fd1\u63a5\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316</a>\uff08PPO\uff09<a href=\"https://pytorch.org\">\u306ePyTorch\u5b9f\u88c5\u3067\u3059</a>\u3002</p>\n<p>PPO\u306f\u5f37\u5316\u5b66\u7fd2\u306e\u30dd\u30ea\u30b7\u30fc\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u6cd5\u3067\u3059\u3002\u5358\u7d14\u306a\u30dd\u30ea\u30b7\u30fc\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30e1\u30bd\u30c3\u30c9\u3067\u306f\u3001\u30b5\u30f3\u30d7\u30eb\uff08\u307e\u305f\u306f\u30b5\u30f3\u30d7\u30eb\u306e\u30bb\u30c3\u30c8\uff09\u3054\u3068\u306b1\u3064\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u66f4\u65b0\u3092\u884c\u3044\u307e\u3059\u30021\u3064\u306e\u30b5\u30f3\u30d7\u30eb\u306b\u5bfe\u3057\u3066\u8907\u6570\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30b9\u30c6\u30c3\u30d7\u3092\u5b9f\u884c\u3059\u308b\u3068\u3001\u30dd\u30ea\u30b7\u30fc\u306e\u504f\u5dee\u304c\u5927\u304d\u3059\u304e\u3066\u4e0d\u9069\u5207\u306a\u30dd\u30ea\u30b7\u30fc\u304c\u751f\u6210\u3055\u308c\u308b\u305f\u3081\u3001\u554f\u984c\u304c\u767a\u751f\u3057\u307e\u3059\u3002PPO \u3067\u306f\u3001\u30dd\u30ea\u30b7\u30fc\u3092\u30c7\u30fc\u30bf\u306e\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306b\u4f7f\u7528\u3057\u305f\u30dd\u30ea\u30b7\u30fc\u306b\u8fd1\u3044\u72b6\u614b\u306b\u4fdd\u3064\u3053\u3068\u3067\u3001\u30b5\u30f3\u30d7\u30eb\u3054\u3068\u306b\u8907\u6570\u306e\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u66f4\u65b0\u3092\u884c\u3046\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002\u66f4\u65b0\u3055\u308c\u305f\u30dd\u30ea\u30b7\u30fc\u304c\u30c7\u30fc\u30bf\u306e\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u306b\u4f7f\u7528\u3055\u308c\u305f\u30dd\u30ea\u30b7\u30fc\u306b\u5408\u308f\u306a\u3044\u5834\u5408\u306f\u3001\u30b0\u30e9\u30c7\u30fc\u30b7\u30e7\u30f3\u30d5\u30ed\u30fc\u3092\u30af\u30ea\u30c3\u30d4\u30f3\u30b0\u3057\u3066\u66f4\u65b0\u3057\u307e\u3059</p>\u3002\n<p><a href=\"https://nn.labml.ai/rl/ppo/experiment.html\">\u3053\u308c\u3092\u4f7f\u3063\u305f\u5b9f\u9a13\u306f\u3053\u3061\u3089\u304b\u3089\u3054\u89a7\u3044\u305f\u3060\u3051\u307e\u3059</a>\u3002\u3053\u306e\u5b9f\u9a13\u3067\u306f\u3001<a href=\"https://nn.labml.ai/rl/ppo/gae.html\">\u4e00\u822c\u5316\u30a2\u30c9\u30d0\u30f3\u30c6\u30fc\u30b8\u63a8\u5b9a\u3092\u4f7f\u7528\u3057\u3066\u3044\u307e\u3059</a></p>\u3002\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"Proximal Policy Optimization - PPO": "\u8fd1\u63a5\u30dd\u30ea\u30b7\u30fc\u6700\u9069\u5316-PPO"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"<h1><a href=\"https://nn.labml.ai/rl/ppo/index.html\">Proximal Policy Optimization - PPO</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of <a href=\"https://arxiv.org/abs/1707.06347\">Proximal Policy Optimization - PPO</a>.</p>\n<p>PPO is a policy gradient method for reinforcement learning. Simple policy gradient methods one do a single gradient update per sample (or a set of samples). Doing multiple gradient steps for a singe sample causes problems because the policy deviates too much producing a bad policy. PPO lets us do multiple gradient updates per sample by trying to keep the policy close to the policy that was used to sample data. It does so by clipping gradient flow if the updated policy is not close to the policy used to sample the data.</p>\n<p>You can find an experiment that uses it <a href=\"https://nn.labml.ai/rl/ppo/experiment.html\">here</a>. The experiment uses <a href=\"https://nn.labml.ai/rl/ppo/gae.html\">Generalized Advantage Estimation</a>.</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a> </p>\n": "<h1><a href=\"https://nn.labml.ai/rl/ppo/index.html\">\u8fd1\u7aef\u7b56\u7565\u4f18\u5316-PPO</a></h1>\n<p>\u8fd9\u662f P <a href=\"https://pytorch.org\">yTorch</a> \u5b9e\u73b0\u7684<a href=\"https://arxiv.org/abs/1707.06347\">\u8fd1\u7aef\u7b56\u7565\u4f18\u5316-PPO</a>\u3002</p>\n<p>PPO \u662f\u4e00\u79cd\u7528\u4e8e\u5f3a\u5316\u5b66\u4e60\u7684\u7b56\u7565\u68af\u5ea6\u65b9\u6cd5\u3002\u7b80\u5355\u7684\u7b56\u7565\u68af\u5ea6\u65b9\u6cd5\u53ef\u4ee5\u5bf9\u6bcf\u4e2a\u6837\u672c\uff08\u6216\u4e00\u7ec4\u6837\u672c\uff09\u8fdb\u884c\u4e00\u6b21\u68af\u5ea6\u66f4\u65b0\u3002\u5bf9\u5355\u4e2a\u6837\u672c\u6267\u884c\u591a\u4e2a\u68af\u5ea6\u6b65\u9aa4\u4f1a\u5bfc\u81f4\u95ee\u9898\uff0c\u56e0\u4e3a\u8be5\u7b56\u7565\u504f\u79bb\u5f97\u592a\u5927\uff0c\u4ece\u800c\u4ea7\u751f\u4e86\u9519\u8bef\u7684\u7b56\u7565\u3002PPO \u5141\u8bb8\u6211\u4eec\u5728\u6bcf\u4e2a\u6837\u672c\u4e2d\u8fdb\u884c\u591a\u6b21\u68af\u5ea6\u66f4\u65b0\uff0c\u65b9\u6cd5\u662f\u5c3d\u91cf\u4f7f\u7b56\u7565\u4e0e\u7528\u4e8e\u91c7\u6837\u6570\u636e\u7684\u7b56\u7565\u4fdd\u6301\u4e00\u81f4\u3002\u5982\u679c\u66f4\u65b0\u540e\u7684\u7b56\u7565\u4e0e\u7528\u4e8e\u91c7\u6837\u6570\u636e\u7684\u7b56\u7565\u4e0d\u63a5\u8fd1\uff0c\u5219\u901a\u8fc7\u524a\u51cf\u68af\u5ea6\u6d41\u6765\u5b9e\u73b0\u6b64\u76ee\u7684\u3002</p>\n<p>\u4f60\u53ef\u4ee5<a href=\"https://nn.labml.ai/rl/ppo/experiment.html\">\u5728\u8fd9\u91cc</a>\u627e\u5230\u4e00\u4e2a\u4f7f\u7528\u5b83\u7684\u5b9e\u9a8c\u3002\u8be5\u5b9e\u9a8c\u4f7f\u7528<a href=\"https://nn.labml.ai/rl/ppo/gae.html\">\u5e7f\u4e49\u4f18\u52bf\u4f30\u8ba1</a>\u3002</p>\n<p><a href=\"https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/rl/ppo/experiment.ipynb\"><span translate=no>_^_0_^_</span></a></p>\n",
|
||||
"Proximal Policy Optimization - PPO": "\u8fd1\u7aef\u7b56\u7565\u4f18\u5316-PPO"
|
||||
}
|
||||
Reference in New Issue
Block a user