On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting — part 3/5

Cleaned-text code points [31659, 47651). Bibliography omitted here; source.md includes it.

From the changes in entropy loss, we can observe that by applying ϕ ⁡ ( ⋅ ) \phi(\cdot) , the model maintains a great balance between exploration and exploitation while performing off-policy and on-policy learning simultaneously. On one hand, Chord - ϕ \phi prevents the entropy from collapsing prematurely, which may occur when the SFT loss forces the model to become over-confident on high-probability tokens from the expert data. On the other hand, it avoids large entropy spikes and training instability that may occur if the off-policy expert data drastically conflict with the current policy’s predictions, as the performance curve remains stable throughout the training process. The rewards curve indicates that Chord - ϕ \phi achieves a stable and continuous increase in rewards, resulting in significantly better performance than Pure RL. These results demonstrate that the proposed token-wise weighting function is crucial for effectively unifying the SFT and RL phases.

Tuning μ \mu When Applying Chord - ϕ \phi Empirical observations show that, when ϕ ⁡ ( ⋅ ) \phi(\cdot) is used for fine-grained control over the influence of expert data, a complex and decaying schedule for μ \mu is no longer essential. Chord - ϕ \phi is effective to work with a fixed value for μ \mu (e.g., 0.1 in this study) since it inherently prevents both token-level overfitting and the disruption of established response patterns. The design of ϕ ⁡ ( ⋅ ) \phi(\cdot) simplifies the practical usage of Chord by making it robust to the specific choice of μ \mu . In Appendix B.7 , we provide experiments on tuning the schedule of μ \mu in conjunction with ϕ ⁡ ( ⋅ ) \phi(\cdot) .

Principle for Instantiating ϕ ⁡ ( ⋅ ) \phi(\cdot) It is worth noting that the proposed weight ϕ ⁡ ( ⋅ ) = p t ∗ ( 1 − p t ) \phi(\cdot)=p_{t}*(1-p_{t}) serves as a concrete and interpretable instantiation following a general principle: stabilizing off-policy integration requires down-weighting the learning signal for tokens at both ends of the probability spectrum. This instantiation is computationally efficient, requiring only element-wise multiplication of existing forward-pass probabilities. As grounded in our empirical observations, by assigning negligible weight to tokens that the policy is already certain about (where p t p_{t} is close to 0 or 1), the proposed method prevents off-policy data from disrupting the model’s established reasoning patterns and focuses updates on tokens where the model is still uncertain. Beyond the specific formulation of ϕ ⁡ ( ⋅ ) \phi(\cdot) , this general principle that enables stable and selective learning from off-policy data can potentially inspire more advanced weighting schemes that are suitable for different scenarios.

We also experiment with several variants of the token-weighting function (such as entropy-based variants, clipping variants, and focal loss), with detailed experiment results and discussions presented in Appendix B.2 . We acknowledge that this ϕ ⁡ ( ⋅ ) \phi(\cdot) token-wise weighting design primarily illustrates a generalizable principle. While it is unlikely that a single universally optimal design exists across all tasks, datasets, and models, our empirical results confirm that our design nonetheless serves as a highly effective and robust instantiation.

### 4.4 Further Analysis

Varying Expert Data Source We investigate the effect of using two different expert sources: the powerful DeepSeek-R1, and the weaker Qwen2.5-72B-Instruct, whose response pattern is closer to the LLaMA3.2-3B-Instruct policy model. Experimental results demonstrate that our proposed methods, Chord - μ \mu and Chord - ϕ \phi , outperform Pure RL and SFT+RL baselines regardless of the expert. We also observe that methods which rely more heavily on expert imitation (e.g., SFT+RL and Chord - μ \mu ) can yield greater gains when the expert is stylistically similar to the policy model. This aligns with our insight: the effectiveness of unifying SFT and RL depends not only on expert data quality but also on the degree of pattern shift it introduces. For detailed results and discussion, please refer to Appendix B.3 .

Extending to Non-verifiable Domains To test the generalizability of Chord beyond verifiable tasks, we conduct experiments on RaR-Medicine, a medical question-answering dataset that lacks deterministic verification. The results show that both Chord - μ \mu and Chord - ϕ \phi significantly outperform pure RL. Chord - ϕ \phi achieves faster convergence and higher final rewards, while Chord - μ \mu exhibits a similar “shift-readapt” pattern as observed in the main experiments (Figure 10 in Appendix B.4 ). These findings validate that our approach successfully generalizes to more diverse, non-verifiable domains. Refer to Appendix B.4 for more details.

Training Weaker Policy Models While the effectiveness of on-policy exploration (RL) is often limited for weaker models, our experiments reveal that their capability to absorb knowledge from expert data simultaneously degrades. Our experiment on Qwen2.5-3B-Instruct shows that due to this dual limitation, a weaker model tends more to suffer from a performance collapse when training on the same off-policy expert data. In such a setting, naive imitation fails and a simple SFT+RL combination proves unstable, whereas our Chord - ϕ \phi provides a robust objective to navigate this trade-off and maintain learning stability. We acknowledge that with larger quantities of higher-quality expert data, a weaker model might overcome this imitation bottleneck, potentially making SFT-leaning methods more favorable. Conversely, this also suggests that our method becomes particularly advantageous when dealing with weaker models under limited expert data, highlighting its strong capability to flexibly balance this trade-off and create a robust synergy between SFT and RL. We defer detailed experiment results and analysis to Appendix B.5 .

## 5 Related Works

Recent advancements in RL show significant success in complex reasoning tasks ( Guo et al., 2025 ; Shao et al., 2024 ; Lambert et al., 2024 ) . However, RL-based exploration is often constrained by the model’s initial knowledge, making it difficult for the model to discover superior reasoning pathways ( Yue et al., 2025 ) . Incorporating off-policy expert data into the on-policy RL loop is a promising strategy to address such exploration challenge. Some studies directly mix expert data with self-rollout generations, either through simple dataset mixing ( Li and Khashabi, 2025 ) , or mixing expert trajectories into on-policy rollout groups ( Yan et al., 2025 ; Fu et al., 2025 ) , while others use expert data to guide generation ( Liu et al., 2025a ; Zhang et al., 2025b ; Huang et al., 2025 ) . A third category interleaves RL updates with SFT steps on expert data, either on a predefined or adaptive schedule ( Chen et al., 2025c ) , or for challenging examples ( Ma et al., 2025 ) . More recently, SRFT ( Fu et al., 2025 ) proposed a unified framework that combines data mixing with a sample-level SFT loss. In this study, we focus on tuning an instruct model that already establishes its own response pattern, which can be a more challenging yet practical scenario compared to existing works that finetune a base model ( Yan et al., 2025 ; Fu et al., 2025 ) . For a more comprehensive literature review, please refer to Appendix C .

## 6 Conclusions and future directions

In this study, we identify that the sequential SFT-then-RL paradigm often yields suboptimal performance by disrupting established patterns with off-policy expert data. To address this, we unify SFT and RL through the lens of on-policy versus off-policy learning and propose Chord . By analyzing the influence of expert data at both the holistic and granular levels, Chord first integrates a global coefficient μ \mu to manage off-policy expert influence, enabling a smoother transition from imitation to exploration. Chord then introduces a token-wise function ϕ ⁡ ( ⋅ ) \phi(\cdot) to selectively absorb expert knowledge by down-weighting tokens that are either already highly probable or extremely improbable. Extensive experiments demonstrate that this unified approach achieves notable improvements over existing sequential pipelines and various baselines.

Looking forward, our work highlights several avenues for future research. First, given the inherent high variance and parameter sensitivity typical of RL processes, exploring further algorithmic advancements to fully stabilize the training dynamics represents a promising direction. Specifically within our framework, since configurations for μ \mu and ϕ ⁡ ( ⋅ ) \phi(\cdot) can vary across setups, discovering more effective adaptive tuning functions to reduce hyperparameter dependence remains a practical objective. Furthermore, while our analysis of pattern shifts is primarily empirical, a deeper understanding of how diverse CoT patterns influence learning would provide valuable insights for tracking training dynamics. Finally, we envision extending this unified paradigm to incorporate heterogeneous mixtures of expert data sources, enabling a comprehensive study of how different model-generated reasoning patterns emerge and evolve throughout the training process. We hope our work inspires the community to continue refining unified post-training paradigms across a broader spectrum of applications.

## Acknowledgements

We would like to thank Qi Liu, Zexi Li, and Ao Li for their suggestions. We also thank the anonymous reviewers and Area Chairs for their constructive feedback during the review process.

## Reproducibility Statement

We release our implementation to inspire further research.

## Appendix A Experimental Setups

### A.1 Hyperparameters

Across all experiments, we adopt the Adam optimizer with β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.999 \beta_{2}=0.999 . The learning rate is tuned within { 1 × 10 − 6 , 5 × 10 − 6 , 1 × 10 − 5 } \{1\times 10^{-6},5\times 10^{-6},1\times 10^{-5}\} , and the temperature for both rollout and evaluation is 1.0. The max response length is set to 16k tokens. For SFT, we train for a maximum of 3 epochs. For RL, we employ “strict on-policy training” similar to ( Liu et al., 2025b ) , where we generate K = 8 K=8 rollouts per prompt before each policy update.

For mathematical reasoning problems, the batch size for SFR/RL is 64/32, and the maximum number of RL steps is 1,500. For tool-use tasks, the batch size is 96 for both RL and SFT, and the maximum number of RL steps is 100. The μ \mu decay schedule is to decrease from 0.9 to 0.05 over the first 30 training steps.

### A.2 Implementation Details

In our experiments, the reward function is tailored to the task-specific requirements. For mathematical reasoning problems, we use a hierarchical reward scheme to encourage both correctness and format adherence. To guarantee the precision of our correctness evaluation, we exclusively sample problems that have integer answers when preparing our dataset. A response receives a reward of + 1.0 +1.0 for a correct final answer. If the format is correct (e.g., step-by-step reasoning ending with a boxed answer) but the answer is wrong, it receives a neutral reward of 0.0 0.0 . A small penalty of − 0.1 -0.1 is applied for responses that are both factually incorrect and improperly formatted. Finally, we penalize overly long and inconclusive responses ( Yu et al., 2025 ) , and apply a strong penalty of − 1.0 -1.0 for exceeding the predefined token limit without a final answer. For tool-use tasks, we employ a simpler binary reward. A response is given a reward of + 1.0 +1.0 if it is completely correct, and 0.0 0.0 otherwise.

We implement SFT algorithms based on LLaMA-Factory ( Zheng et al., 2024 ) , and implement RL algorithms based on Trinity-RFT ( Pan et al., 2025 ) . Experiments are conducted on 8 NVIDIA A100 GPUs and 8 NVIDIA H20 GPUs.

For evaluation, we adopt accuracy as the metric. To avoid high variance in results and ensure fair comparisons, we report avg@32 on AIME24 and AIME 25, and avg@8 on AMC, respectively. Reported results are on the best checkpoint determined by the validation set.

### A.3 Prompts

#### Prompt for Math Problems

The adopted prompt for math problems is shown below.

For the performance of the base model, we report the higher score achieved using either the above prompts for math problems or the default prompt provided by Qwen ( Yang et al., 2024b ) : “Please reason step by step, and put your final answer within \boxed{} ”.

#### Prompt for the MMLU-Pro Dataset

The adopted prompt for the MMLU-Pro dataset is shown below. We use the same system prompt as for the math problems, except that for multiple-choice questions, we modify the answer format to require the corresponding integer as the response.

#### Prompt for the Tool-use Tasks

For the tool-use tasks, we follow ( Zhang et al., 2025a ) to adopt their experimental setup and use the prompt provided in their Figure 8. This prompt is consistently applied to train the LLaMA3.2-3B-Instruct policy model and to generate SFT data with the DeepSeek-R1 expert model.

## Appendix B Experimental Results and Analysis

### B.1 Adaptive Tuning μ \mu

In addition to the fixed decay schedule for μ \mu , we explored an adaptive strategy to dynamically adjust the SFT loss weight based on the model’s ongoing performance, as measured by the average reward. We conducted experiments to validate this idea.

On the Tool-use task, we implemented a strategy where μ \mu is adjusted based on the mean reward of the rollouts. Specifically, for a given reward threshold τ \tau , the new μ \mu is calculated as μ ′ = max ⁡ ( 0 , τ − reward_mean ) \mu^{\prime}=\max(0,\tau-\text{reward\_mean}) . This mechanism ensures that as the model’s average reward surpasses the threshold, the SFT component is gradually phased out ( μ ′ → 0 \mu^{\prime}\to 0 ), allowing the training to focus purely on RL. We tested this with thresholds τ = 0.5 \tau=0.5 and τ = 0.7 \tau=0.7 .

The results, presented in Table 3 , show that setting the reward threshold to 0.5 yields a strong overall score of 78.1, which is highly competitive with our main approach using a fixed decay schedule. This indicates that dynamically reducing the SFT contribution as the model improves is a viable and effective strategy.

However, we also found that this configuration is still dependent on task-specific hyperparameter tuning. When we set a higher threshold of 0.7, performance degraded significantly. This is likely because the policy was subjected to excessive SFT even when achieving moderately high rewards, disrupting the optimization process.

These experiments serve as a proof of concept, demonstrating that an automated, reward-aware schedule for μ \mu can work effectively and represents a logical extension of our core ideas. However, since this method still requires tuning another hyperparameter(the reward threshold), its practical implementation is not necessarily simpler to tune. Therefore, we present this as a preliminary exploration into adaptive mixing coefficients, leaving a more thorough investigation of robust and generalizable adaptive schemes as a promising direction for future work.

### B.2 Varying the ϕ \phi Function

To validate the robustness of our approach and explore alternative weighting strategies, we conduct ablation studies comparing different variants of the token-wise weighting function ϕ ⁡ ( ⋅ ) \phi(\cdot) against our proposed method. We evaluate these variants on both the tool-use and mathematical reasoning tasks.

#### Evaluated ϕ \phi Variants.

We compare the following token-wise weighting strategies:

• Chord - ϕ \phi (Ours) : Our proposed method, with ϕ ⁡ ( p ) = p × ( 1 − p ) \phi(p)=p\times(1-p) .

• Entropy Top : Only trains on the top 5% of tokens with the highest entropy, setting ϕ ⁡ ( ⋅ ) = 1 \phi(\cdot)=1 for these tokens and ϕ ⁡ ( ⋅ ) = 0 \phi(\cdot)=0 for others.