Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective — part 8/8

Cleaned-text code points [109441, 116514). Bibliography omitted here; source.md includes it.

##### Part 1

In Part 1 ( Qin et al., 2024 ) , the tree data generated through beam search is traversed, with specific nodes refined by GPT-4 and then used in supervised fine-tuning. The examples presented in the paper highlight the model’s self-reflection capabilities, which comes from the refinement of GPT-4. The approach adopted in Part 1 can be described as expert iteration, where SFT is applied to the data generated via search. Part 1 also compares PRM from o1-mini anotation versus Math-Shepherd ( Wang et al., 2024c ) , they found that o1-mini outperforms Math-Shepherd.

##### Part 2

Part 2 ( Huang et al., 2024b ) of the o1 journey introduces a radically different approach. While Part 1 focuses on reinforcement learning, Part 2 attempts to distill o1-mini. Although o1-mini conceals the chain-of-thought (CoT) and only outputs the summary of CoT, Part 2 tries to recover the hidden CoT by prompt o1-mini to augment the summary. Through distillation, they found that Qwen-72B outperforms o1-preview on AIME. However, it does not mean that distillation enables the student model to outperform the teacher model, since o1-mini also surpasses o1-preview on AIME.

### Open-Reasoner

The framework of Open-Reasoner ( Wang et al., 2024b ) is akin to AlphaGo ( Silver et al., 2016 ) , utilizing reinforcement learning to enhance the model’s performance. During the testing phase, Monte Carlo Tree Search (MCTS) is employed to identify optimal solutions. This search algorithm is applied exclusively during testing, while the training data is derived by sampling with the current policy. Additionally, Open-Reasoner employs a method similar to Math-Shepherd ( Wang et al., 2024c ) for training the reward model.

### Slow Thinking with LLMs

Similarly to the o1 journey, Slow Thinking with LLMs is alos outlined in two technical reports ( Jiang et al., 2024b ; Min et al., 2024 ) .

##### Part 1

The part 1 of Slow Thinking with LLMs ( Jiang et al., 2024b ) is similar to OpenReasoner ( Wang et al., 2024b ) , incorporating both reinforcement learning and test-time search. But unlike Open-Reasoner, it employs the DPO instead of PPO algorithm during training. In the testing phase, it also employs the MCTS algorithm for search.

##### Part 2

The part 2 of Slow Thinking with LLMs ( Min et al., 2024 ) distills from QwQ ( Team, 2024 ) and Deepseek-R1 ( Shao et al., 2024 ) and tries two methods for reinforcement learning: DPO and SFT on data generated from reject sampling. They found that distilling on thousands of examples from QwQ and Deepseek-R1 can significantly bring a substantial improvement on challenging reasoning tasks and reinforcement learning can bring further improvement based on distillation.

### Marco-o1

Marco-o1 ( Zhao et al., 2024a ) integrates the data from Open-o1 with data generated by the model itself through the MCTS algorithm for SFT training. Marco-o1 demonstrates that prompting the model for self-reflection after each step of the MCTS process enhances the effectiveness of the search.

### o1-coder

o1-coder ( Zhang et al., 2024g ) attempts to reimplement o1 on the code generation. They train a generator to generate test case for providing outcome reward. With outcome rewards, they use MCTS algorithm to generate code solutions, which are then used for improving policy model through SFT. Their train a PRM following Wang et al. (2024c) which is updated with the improvement of the policy.

## 8 Future Directions

##### How to adapt o1 to general domains?

The key to adapt o1 to general domains is the availability of a general reward models. How can we learn a general reward model for general domains? For reasoning tasks, there is typically a standard answer, allowing for the training of an outcome reward model. Using the reward shaping techniques introduced in Section 4 , a process reward model can be further trained on top of the outcome reward model. For non-reasoning tasks, such as alignment tasks, obtaining an outcome reward is often challenging. In such cases, methods introduced in Section 4 , such as learning from feedback, become crucial. A Bradley-Terry model ( Leike et al., 2018 ) can be trained from preference data, or a reward model can be developed from expert data using inverse reinforcement learning.

##### How to introduce multiple modality to o1?

The challenge of incorporating multiple modality into o1 primarily lies in the alignment or grounding between text and other modalities. For example, when the image modality is introduced, the model needs to handle the fine-grained relationships between long CoT and images. When CoT becomes lengthy, establishing such connections becomes more difficult. Therefore, Gao et al. (2024) incorporated the image modality into the CoT, so that the CoT generated by the model includes both text and images. This approach enhances the fine-grained connections between the generated text and images, but it also introduces new challenges. Introducing information from the image or other modalities into the COT significantly increases its length, leading to increased inference latency. A potential solution to this issue could be to use continious representations to generate CoT, replacing textual and other modality-specific information.

##### How to learn and search with a world model?

OpenAI has outlined a 5-stage roadmap to AGI, with stage 2 focusing on becoming a strong reasoner, stage 3 centered on becoming an agent.The o1 has already reached stage 2, achieving reasoning capabilities on par with human experts. Consequently, o1’s next goal is to progress to stage 3—being able to take actions in real environment and solving real-environment tasks.

The key to extending o1 to real-world environments lies in reward modeling, more specifically in environment or world modeling. This concept of world models was also introduced in Section 4 . When a simulator of the real environment is established, the agent can interact with the world model rather than directly with the environment. The world model plays a vital role in both training and testing the agent. During training, interacting with the world model is more efficient than direct interaction with the environment. During testing, the agent can leverage the world model for planning or searching, identifying the optimal strategy before executing actions in the real environment. Without a world model, it is impossible for the model to perform search or planning, as the real environment is not time-reversible. For example, in the game of Go, once a move is made, it cannot be undone.

## 9 Conclusion

In this paper, we present a roadmap for reproducing o1 from the perspective of reinforcement learning, emphasizing key components such as policy initialization, reward design, search, and learning. We offer a comprehensive survey of these components and show that existing open-source projects that attempt to reproduce o1 are variations of our roadmap. Finally, we hope this roadmap inspire further research on overcoming the challenges involved in reproducing o1.