1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

arxiv:2503.14858

Original source · Paper JSON

Reinforcement learning · 2025-03-19

Cites (5) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
• Abstract
• 1 Introduction
• 2 Related Work
• 3 Preliminaries
• Goal-Conditioned Reinforcement Learning
• Contrastive Reinforcement Learning.
• Residual Connections
• 4 Experiments
• 4.1 Experimental Setup
• Environments.
• Architectural Components
• 4.2 Scaling Depth in Contrastive RL
• 4.3 Emergent Policies Through Depth
• 4.4 What Matters for CRL Scaling
• Width vs. Depth
• Scaling the Actor vs. Critic Networks
• Deep Networks Unlock Batch Size Scaling
• Training Contrastive RL with 1000+ Layers
• 4.5 Why Scaling Happens
• Depth Enhances Contrastive Representations
• Depth Enhances Exploration and Expressivity in a Synergized Way
• Deep Networks Learn to Allocate Greater Representational Capacity to States Near the Goal
• Deeper Networks Enable Partial Experience Stitching
• The (CRL) Algorithm is Key
• 4.6 Does Depth Scaling Improve Offline Contrastive RL?
• 5 Conclusion
• Limitations.
• Impact Statement
• Acknowledgments.
• Appendix A Additional Experiments
• A.1 Scaled CRL Outperforms All Other Baselines on 8 out of 10 Environments
• A.2 The CRL Algorithm is Key: Depth Scaling is Not Effective on Other Baselines
• A.3 Additional Scaling Experiments: Offline GCBC, BC, and QRL
• A.4 Can Depth Scaling also be Effective for Quasimetric Architectures?
• A.5 Additional Architectural Ablations: Layer Norm and Swish Activation
• A.6 Can We Integrate Novel Architectural Innovations from the Emerging RL Scaling Literature?
• A.7 Residuals Norms in Deep Networks
• A.8 Scaling Depth for Offline Goal-conditioned RL
• Appendix B Experimental Details
• B.1 Environment Setup and Hyperparameters
• B.2 Python Environment Differences
• B.3 Wall-clock Time of Our Approach
• B.4 Wall-clock Time: Comparison to Baselines
• NeurIPS Paper Checklist
• Instructions for reporting errors