Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

arxiv:1801.01290

Original source · Paper JSON

Reinforcement learning · 2018-01-04

Cites (1) · Cited by (10)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Work
• 3 Preliminaries
• 3.1 Notation
• 3.2 Maximum Entropy Reinforcement Learning
• 4 From Soft Policy Iteration to Soft Actor-Critic
• 4.1 Derivation of Soft Policy Iteration
• Lemma 1 (Soft Policy Evaluation) .
• Proof.
• Lemma 2 (Soft Policy Improvement) .
• Proof.
• Theorem 1 (Soft Policy Iteration) .
• Proof.
• 4.2 Soft Actor-Critic
• 5 Experiments
• 5.1 Comparative Evaluation
• 5.2 Ablation Study
• Stochastic vs. deterministic policy.
• Policy evaluation.
• Reward scale.
• Target network update.
• 6 Conclusion
• Acknowledgments
• Appendix A Maximum Entropy Objective
• Appendix B Proofs
• B.1 Lemma 1
• Proof.
• B.2 Lemma 2
• Proof.
• B.3 Theorem 1
• Proof.
• Appendix C Enforcing Action Bounds
• Appendix D Hyperparameters
• Appendix E Additional Baseline Results
• Instructions for reporting errors