In-Context Reinforcement Learning for Variable Action Spaces

arxiv:2312.13327

Original source · Paper JSON

Reinforcement learning · 2023-12-20

Cites (0) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• In-Context Reinforcement Learning for Variable Action Spaces
• Abstract
• Keywords:
• 1 Introduction
• 2 Algorithm Distillation Struggles with Novel Action Spaces
• 3 Headless-AD
• 4 Experiments
• 4.1 Bernoulli Bandit
• 4.2 Contextual Bandit
• 4.3 Darkroom
• 5 Ablations
• 5.1 Action Set Prompt
• 5.2 Contrastive Loss
• 5.3 Orthonormal Action Embeddings
• 6 Related Work
• 7 Conclusion
• Impact Statement
• Appendix A Background
• Appendix B Related Work
• B.1 Transformers in Reinforcement Learning
• B.2 Offline Meta-RL
• B.3 In-Context Learning in RL
• B.4 Discarding the Linear Layer
• Appendix C Algorithm Distillation on Permuted Train Sets
• Appendix D Across-Environment Generalization
• Appendix E Visual Darkroom
• Appendix F Darkroom. Alternative Split
• Appendix G MSE-Headless-AD In-Context Curves on Bernoulli Bandit
• Appendix H Sampling of Orthonormal Vectors
• Appendix I Algorithms’ Training Times
• Appendix J Model Hyperparameters
• Appendix K Linear Dependence of Different Types of Action Embeddings
• Appendix L Code Sample
• Instructions for reporting errors