Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

arxiv:2005.01643

Original source · Paper JSON

Reinforcement learning · 2020-05-04

Cites (1) · Cited by (5)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10
11. Part 11
12. Part 12

Contents

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Abstract
• 1 Introduction
• 2 Offline Reinforcement Learning Problem Statement and Overview
• 2.1 Reinforcement Learning Preliminaries
• Definition 2.1 (Markov decision process) .
• Definition 2.2 (Partially observed Markov decision process) .
• Policy gradients.
• Approximate dynamic programming.
• Actor-critic algorithms.
• Model-based reinforcement learning.
• 2.2 Offline Reinforcement Learning
• 2.3 Example Scenarios
• Decision making in health care.
• Learning goal-directed dialogue policies.
• Learning robotic manipulation skills.
• 2.4 What Makes Offline Reinforcement Learning Difficult?
• Theorem 2.1 (Behavioral cloning error bound) .
• Proof.
• Theorem 2.2 (DAgger error bound) .
• Proof.
• 3 Offline Evaluation and Reinforcement Learning via Importance Sampling
• 3.1 Off-Policy Evaluation via Importance Sampling
• 3.2 The Off-Policy Policy Gradient
• 3.3 Approximate Off-Policy Policy Gradients
• 3.4 Marginalized Importance Sampling
• Forward Bellman equation based approaches.
• Backward Bellman equation based approaches via convex duality.
• 3.5 Challenges and Open Problems
• 4 Offline Reinforcement Learning via Dynamic Programming
• 4.1 Off-Policy Value Function Estimation with Linear Value Functions
• Bellman residual minimization.
• Least-squares fixed point approximation.
• Least squares temporal difference Q-learning (LSTD-Q).
• Least squares policy iteration (LSPI).
• 4.2 Distributional Shift in Offline Reinforcement Learning via Dynamic Programming
• 4.3 Policy Constraints for Off-Policy Evaluation and Improvement
• Explicit f f -divergence constraints.
• Implicit f f -divergence constraints.
• Integral probability metrics (IPMs).
• Tradeoffs between different constraint types.
• 4.4 Offline Approximate Dynamic Programming with Uncertainty Estimation
• 4.5 Conservative Q-Learning and Pessimistic Value Functions
• 4.6 Challenges and Open Problems
• 5 Offline Model-Based Reinforcement Learning
• 5.1 Model Exploitation and Distribution Shift
• 5.2 Brief Survey of Model-Based Offline Reinforcement Learning
• Offline RL with standard model-based RL methods.
• Off-policy evaluation with models.
• Distribution and safe region constraints.
• Conservative model-based RL algorithms.
• 5.3 Challenges and Open Problems
• 6 Applications and Evaluation
• 6.1 Evaluation and Benchmarks
• 6.2 Applications in Robotics
• 6.3 Applications in Healthcare
• 6.4 Applications in Autonomous Driving
• 6.5 Applications in Advertising and Recommender Systems
• 6.6 Applications in Language and Dialogue
• 7 Discussion and Perspectives