A Distributional Perspective on Reinforcement Learning

arxiv:1707.06887

Original source · Paper JSON

Reinforcement learning · 2017-07-21

Cites (3) · Cited by (9)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• A Distributional Perspective on Reinforcement Learning
• Abstract
• 1 Introduction
• 2 Setting
• 2.1 Bellman’s Equations
• 3 The Distributional Bellman Operators
• 3.1 Distributional Equations
• 3.2 The Wasserstein Metric
• Lemma 1 (Partition lemma) .
• Lemma 2 .
• 3.3 Policy Evaluation
• 3.3.1 Contraction in d ¯ p \bar{d}_{p}
• Lemma 3 .
• 3.3.2 Contraction in Centered Moments
• 3.4 Control
• Definition 1 (Optimal value distribution) .
• Definition 2 .
• Lemma 4 .
• Definition 3 .
• Theorem 1 (Convergence in the control setting) .
• Proposition 1 .
• Proposition 2 .
• Proposition 3 .
• 4 Approximate Distributional Learning
• 4.1 Parametric Distribution
• 4.2 Projected Bellman Update
• 5 Evaluation on Atari 2600 Games
• 5.1 Varying the Number of Atoms
• 5.2 State-of-the-Art Results
• 6 Discussion
• 6.1 Why does learning a distribution matter?
• Acknowledgements
• Erratum
• Appendix A Related Work
• Appendix B Proofs
• Lemma 1 (Partition lemma) .
• Proof.
• Lemma 2 .
• Proof.
• Lemma 3 .
• Proof.
• Proposition 1 (Sobel, 1982) .
• Proof.
• Lemma 4 .
• Proof.
• Theorem 1 (Convergence in the control setting) .
• Lemma 5 .
• Proof.
• Lemma 6 .
• Proof.
• Proof of Theorem 1 .
• Proposition 4 .
• Lemma 7 (Sample Wasserstein distance) .
• Proof.
• Proposition 5 .
• Appendix C Algorithmic Details
• Appendix D Comparison of Sampled Wasserstein Loss and Categorical Projection
• Appendix E Supplemental Videos and Results
• Instructions for reporting errors