Addressing Function Approximation Error in Actor-Critic Methods

arxiv:1802.09477

Original source · Paper JSON

Reinforcement learning · 2018-02-26

Cites (4) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Supplementary Material
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Work
• 3 Background
• 4 Overestimation Bias
• 4.1 Overestimation Bias in Actor-Critic
• 4.2 Clipped Double Q-Learning for Actor-Critic
• 5 Addressing Variance
• 5.1 Accumulating Error
• 5.2 Target Networks and Delayed Policy Updates
• 5.3 Target Policy Smoothing Regularization
• 6 Experiments
• 6.1 Evaluation
• 6.2 Ablation Studies
• 7 Conclusion
• Appendix A Proof of Convergence of Clipped Double Q-Learning
• Appendix B Overestimation Bias in Deterministic Policy Gradients
• Appendix C DDPG Network and Hyper-parameter Comparison
• Appendix D Additional Implementation Details
• Appendix E Soft Actor-Critic Implementation Details
• Appendix F Additional Learning Curves
• Instructions for reporting errors