Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective

arxiv:2412.14135

Original source · Paper JSON

Reasoning and the "physics" of language models · 2024-12-18

Cites (12) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8

Contents

• Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective
• Abstract
• 1 Introduction
• Policy Initialization
• Reward Design
• Search
• Learning
• 2 Background
• Agent
• Environment
• 3 Policy Initialization
• 3.1 Pre-Training
• 3.1.1 Language Understanding and Generation
• 3.1.2 World Knowledge Acquisition and Storage
• 3.1.3 Basic Reasoning Abilities
• 3.2 Instruction Fine-Tuning
• 3.3 Human-like Reasoning Behaviours
• Problem Analysis
• Task Decomposition
• Task Completion
• Alternative Proposal
• Self-Evaluation
• Self-Correction
• 3.4 Speculation About the Policy Initialization of o1
• Long-Text Generation Capabilities
• Shaping Human-like Reasoning Behaviors Logically
• Self-Reflection
• 3.5 Challenges of Policy Initialization for Reproducing o1
• How to balance sampling-efficiency and sampling-diversity?
• How to ensure domain generalization of reasoning behaviors?
• 4 Reward Design
• 4.1 Outcome Reward vs Process Reward
• 4.2 Reward Design Methods
• 4.2.1 Reward from Environment
• From Realistic Environment
• From Simulating Environment
• From AI Judgment
• 4.2.2 Reward Modeling from Data
• Learning Rewards from Preference Data
• Learning Rewards from Expert Data
• 4.2.3 Reward Shaping
• 4.3 Speculation About the Reward Design of o1
• 4.4 Challenges of Reward Design for Reproducing o1
• How to overcome distribution shift?
• How to design fine-grained reward for language model?
• How to choose data when modeling reward for complex tasks?
• 4.5 Generalization
• 4.5.1 Reward Ensemble
• 4.5.2 One for all: World Model
• 5 Search
• 5.1 The role of Search in o1
• 5.2 Search Guidance
• 5.2.1 Internal Guidance
• Model Uncertainty
• Self-evaluation
• 5.2.2 External Guidance
• Environmental Feedback
• Heuristic Rules
• 5.2.3 Comparison of Internal and External Guidance
• 5.2.4 Combination of Internal and External Guidance
• 5.3 Search Strategies
• 5.3.1 Tree Search
• Best-of-n Sampling
• Beam Search
• Monte Carlo Tree Search
• Other Tree Search Algorithms.
• 5.3.2 Sequential Revisions
• 5.3.3 Comparison of Tree Search and Sequential Revisions
• 5.3.4 Combination of Tree Search and Sequential Revisions
• 5.4 Speculation About the Search of o1.
• Train-time search.
• Test-time search.
• 5.5 Scaling Law of Search
• 5.6 Challenges of Search for Reproducing o1
• How to overcome Inverse Scaling
• How to avoid over-thinking on simple tasks?
• How to trade off tree search and sequential revisions?
• How to improve efficiency of Search?
• 6 Learning
• 6.1 Learning Methods
• 6.1.1 Policy Gradient
• REINFORCE
• Actor-Critic
• PPO
• DPO
• 6.1.2 Behavior Cloning
• 6.1.3 Speculation about the learning of o1
• Memory Cost
• Data Utilization
• 6.2 Scaling Law of Reinforcement Learning
• 6.3 Challenges of Learning for Reproducing o1
• How to improve Training Efficiency?
• How to learn a Strong Question Generator.
• How to Narrow the Distribution Shift in off-policy learning?
• 7 Open-source o1 Project
• g1
• Thinking Claude
• Open-o1
• o1 Journey
• Part 1
• Part 2
• Open-Reasoner
• Slow Thinking with LLMs
• Part 1
• Part 2
• Marco-o1
• o1-coder
• 8 Future Directions
• How to adapt o1 to general domains?
• How to introduce multiple modality to o1?
• How to learn and search with a world model?
• 9 Conclusion