On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting — part 2/5
Cleaned-text code points [15869, 31659). Bibliography omitted here; source.md includes it.
Beyond the Loss Coefficient μ \mu Empirical comparisons (refer to Section 4 for more details) demonstrate that applying a decay schedule to μ \mu yields notable performance gains over the SFT-then-RL paradigm. At the same time, two key observations motivate us to extend beyond μ \mu .
Firstly, as shown in Figure 5 , the learning curve still reveals a “shift-readapt” progress, where the reward initially declines before subsequently increasing. These observations indicate that, despite improvements in performance, learning from off-policy expert data might still disrupt established patterns and stifle the model’s capacity for genuine exploration during on-policy training.
Secondly, the response patterns of the model trained with Chord - μ \mu (as shown in Appendix E ) appear to converge to those of the expert model. Case studies reveal that Chord - μ \mu compels the model to adopt the expert’s verbose response pattern wholesale, hence overwriting its own inherent conciseness. This indicates that while μ \mu controls the overall influence of expert data, it lacks fine-grained precision. As a result, it forces the model to indiscriminately adopt expert patterns, which can create conflicts with its own established style.
Towards the goal of utilizing off-policy data as an incentive and guidance for the model to explore novel and effective reasoning paths, rather than merely as a target to imitate, we further integrate Chord with a token-wise, fine-grained weighting function ϕ ( ⋅ ) \phi(\cdot) , forming a dual-control mechanism together with the global coefficient μ \mu for controlling the influence of the off-policy expert data.
### 3.3 Enhancing the Stability of Off-policy Learning via ϕ ( ⋅ ) \phi(\cdot)
A feasible solution for controlling the influence of off-policy expert data from a fine-grained perspective is to differentiate the tokens based on their generation probabilities π ( y t ∗ | x , y < t ∗ ) \pi(y^{*}_{t}|x,y^{*}_{<t}) . For example, Importance Sampling (IS) ( Schulman et al., 2017 ) has been widely used for stably integrating off-policy data in RL, which suggests re-weighting the objective by the probability ratio between the target policy π θ \pi_{\theta} and the behavior policy π sample \pi_{\text{sample}} that generated the expert data. Formally, the objective function can be given as: ℒ SFT-IS ( θ ) = 𝔼 ( x , y ∗ ) ∼ 𝒟 SFT [ − ∑ t = 1 | y ∗ | sg ( π θ ( y t ∗ | x , y < t ∗ ) π sample ( y t ∗ | x , y < t ∗ ) ) ⋅ log π θ ( y t ∗ | x , y < t ∗ ) ] , \mathcal{L}_{\text{SFT-IS}}(\theta)=\mathbb{E}_{(x,y^{*})\sim\mathcal{D}_{\text{SFT}}}\left[-\sum_{t=1}^{|y^{*}|}\text{sg}\left(\frac{\pi_{\theta}(y^{*}_{t}|x,y^{*}_{<t})}{\pi_{\text{sample}}(y^{*}_{t}|x,y^{*}_{<t})}\right)\cdot\log\pi_{\theta}(y^{*}_{t}|x,y^{*}_{<t})\right], (4) where sg ( ⋅ ) \text{sg}(\cdot) denotes the stop-gradient operator. Note that the probabilities π sample ( y t ∗ | … ) \pi_{\text{sample}}(y^{*}_{t}|\dots) for the expert data 𝒟 SFT \mathcal{D}_{\text{SFT}} are often unknown. Following the common practice ( Yan et al., 2025 ; Wu et al., 2025 ) , we assume that the denominator is 1 1 , treating the expert data as the ground-truth distribution.
From a token-wise perspective, IS enhances training stability by down-weighting low-probability tokens that could disrupt the established policy. As empirical observations shown in Figure 5 , mixing off-policy data without IS leads to a sharp rise in entropy, which implies that the model’s established patterns are quickly disrupted by the unweighted off-policy data. However, we notice that IS can lead to a sharp collapse in policy entropy compared to pure RL, which implies that it can limit the exploration essential for the RL phase and trap the model in a stable but suboptimal solution. The underlying reason is that IS prevents disruptive shifts in the policy distribution by down-weighting low-probability tokens, but it also aggressively reinforces existing high-probability tokens while ignoring novel but low-probability ones, thus causing the policy to become overconfident.
Stabilize Off-policy Data Training with ϕ ( ⋅ ) \phi(\cdot) To tackle this, we propose a fine-grained, per-token weighting function ϕ ( y t ∗ , π θ ) \phi(y^{*}_{t};\pi_{\theta}) that down-weights the learning signal for tokens at both ends of the probability spectrum , i.e., down-weighting those tokens that are highly probable (to prevent entropy collapse) or extremely improbable (to avoid disruption). More specifically, the weight for a given expert token is defined based on the policy’s probability p t = π θ ( y t ∗ | x , y < t ∗ ) p_{t}=\pi_{\theta}(y^{*}_{t}|x,y^{*}_{<t}) , as follows: ϕ ( y t ∗ , π θ ) = p t ( 1 − p t ) , \phi(y^{*}_{t};\pi_{\theta})=p_{t}(1-p_{t}), (5) which naturally forms a parabolic curve that peaks at p t = 0.5 p_{t}=0.5 and decays to zero as p t p_{t} approaches 0 or 1. The SFT objective function can be updated as: ℒ SFT- ϕ ( θ ) = − 𝔼 ( x , y ∗ ) ∼ 𝒟 SFT [ ∑ t = 1 | y ∗ | ϕ ( y t ∗ , π θ ) ⋅ log π θ ( y t ∗ | x , y < t ∗ ) ] , \mathcal{L}_{\text{SFT-}{\phi}}(\theta)=-\mathbb{E}_{(x,y^{*})\sim\mathcal{D}_{\text{SFT}}}\left[\sum_{t=1}^{|y^{*}|}\phi(y^{*}_{t};\pi_{\theta})\cdot\log\pi_{\theta}(y^{*}_{t}|x,y^{*}_{<t})\right], (6) where ϕ ( y t ∗ , π θ ) \phi(y^{*}_{t};\pi_{\theta}) modulates the gradient contribution of each token in the expert trajectory.
From an information-theoretic perspective, the term p t ( 1 − p t ) p_{t}(1-p_{t}) can be viewed as a measure of the policy’s uncertainty ( Wang et al., 2025 ) for the binary event of generating token y t ∗ y^{*}_{t} . Therefore, this approach biases learning towards tokens where the policy is most uncertain, and creates a “learning sweet spot” that focuses the off-policy learning on tokens that are novel enough to be informative but not so divergent as to disrupt the established policy.
By replacing the static ℒ SFT \mathcal{L}_{\text{SFT}} in the proposed hybrid loss function (defined in equation 3 ) with ℒ SFT- ϕ \mathcal{L}_{\text{SFT-$\phi$}} , we obtain the final objective function of Chord , which applies a global coefficient μ \mu for adjusting the overall influence of expert data and a fine-grained weighting function ϕ ( ⋅ ) \phi(\cdot) that helps enhance the stability when learning from off-policy data.
## 4 Experiments
### 4.1 Setup
Datasets, Models, and Evaluations We conduct experiments on mathematical reasoning problems and practical tool-use tasks. (i) For mathematical reasoning problems , we utilize the OpenR1-Math-220k dataset ( Hugging Face, 2025 ) , from which we sample 5k instances for SFT and 20k for RL, ensuring no overlap. Our policy model is Qwen2.5-7B-Instruct, whose response patterns differ significantly from the expert (Deepseek-R1). We evaluate in-domain generalization performance on the AIME24, AIME25, and AMC benchmarks ( Li et al., 2024 ) , and use MMLU-Pro ( Wang et al., 2024 ) to monitor the changes in general reasoning. (ii) For tool-use tasks , we conduct experiments on the single-turn instances of the ToolAce ( Liu et al., 2024 ) dataset. We sample 5k instances for RL and 500 for SFT, for which the expert trajectories are generated by querying the Deepseek-R1 with the same system prompt. We use LLaMA3.2-3B-Instruct ( Grattafiori et al., 2024 ) as our policy model, which also differs in response patterns from the expert (Deepseek-R1). We evaluate the model performance on BFCL ( Patil et al., 2024 ) .
Baselines We compare the proposed Chord with a comprehensive set of baselines, including: (i) Original Model : The original Qwen2.5-7B-Instruct/LLaMA3.2-3B-Instruct model. (ii) SFT-only : The model fine-tuned on the SFT dataset. We focus on two specific configurations: SFT-light , trained for a single epoch, and SFT-best , the peak-performing checkpoint on the test set found by searching over different learning rates and training epochs. (iii) RL-only : The model fine-tuned directly on the RL dataset using the GRPO algorithm. (iv) SFT+RL : The sequential SFT-then-RL paradigm. (v) LUFFY 1 1 1 For math reasoning problems, we utilize 20k samples for training, whereas the original paper utilizes 45k samples and achieves scores of 50.9 on AMC, 17.7 on AIME24, and 14.8 on AIME25. For tool-use tasks, LUFFY utilizes 5k SFT samples instead of 500. ( Yan et al., 2025 ) : A method that integrates expert demonstrations within GRPO rollout groups and reshapes the importance sampling ratio. (vi) SASR ( Chen et al., 2025c ) : A method that probabilistically interleaves SFT and RL steps. It prioritizes SFT when the model’s outputs are dissimilar to expert demonstrations, adapting the training focus dynamically.
For more details of the experimental setups, please refer to Appendix A .
### 4.2 Comparisons
The proposed approaches implemented based on Chord include (i) Chord - μ \mu : We employ a decay schedule for the loss coefficient μ \mu to gradually transition from off-policy to on-policy learning, as detailed in Section 3.2 ; and (ii) Chord - ϕ \phi : We fix the value of μ \mu and further integrate the token-wise weighting function ϕ ( ⋅ ) \phi(\cdot) to achieve a dual-control mechanism on the influence of off-policy expert data, as introduced in Section 3.3 .
Model Performance Overall, the comparisons summarized in Table 1 demonstrate the effectiveness and superiority of Chord on both reasoning problems and tool-use tasks.
Specifically, the experimental results reveal a challenge within the SFT-then-RL paradigm. We notice that minimal tuning on off-policy data (SFT-light) degrades performance, and a more thorough SFT phase (SFT-best) achieves better results. However, the optimal timing for transitioning from SFT to RL can vary across different scenarios. For example, initiating RL from SFT-best yields superior performance on math reasoning problems, while SFT-light+RL performs better on tool-use tasks. This divergence confirms that the SFT-RL balance is highly task-dependent and needs extensive efforts for careful adjustment.
These SFT-then-RL approaches are surpassed by Chord - μ \mu , which enables a smooth transition from off-policy to on-policy learning rather than a rigid switch. Specifically, Chord - μ \mu outperforms the strong SFT-best+RL baseline across all math reasoning benchmarks, achieving improvements of +2.4 on AMC, +1.0 on AIME24, and +1.6 on AIME25, respectively. Besides, Chord - μ \mu also achieves better overall results compared to these SFT-then-RL baselines on tool-use tasks. These results demonstrate the superiority of its unified learning design.
Further, Chord - ϕ \phi achieves consistent outperformance over the baselines. These results demonstrate the effectiveness of our dual-control mechanism in flexibly controlling the influence of off-policy expert data. Chord - ϕ \phi selectively applies the SFT loss to non-disruptive tokens, integrating expert knowledge without compromising foundational abilities. This enables robust learning from both off-policy expert data and on-policy exploration, leading to the best performance on both reasoning problems and tool-use tasks.
Response Patterns We further compare the influence of expert data (generated by DeepSeek-R1) on response patterns across different approaches. As shown in Table 2 , expert responses are substantially longer than the original model’s on both math (6,132 vs. 659 tokens) and tool-use tasks (315 vs. 147 tokens). SFT models (SFT-light and SFT-best) initially mimic this verbosity. However, a subsequent RL can help mitigate the issues of overly lengthy responses by training the models to conduct on-policy exploration. The response length produced by SFT-light+RL is much shorter than that of SFT-best+RL (1,322/119 vs. 4,830/489 tokens), as fewer epochs of SFT allow the model to retain its original response patterns. Besides, from Figure 6 , we can observe that Chord - μ \mu exhibits a similar trend, where the average response length initially increases to align with expert patterns and then gradually converges to a lower length as on-policy training progresses.
On the other hand, Pure RL on instruct-tuned models lengthens math responses (from 659 to 1,423 tokens) while shortening them for tool-use (from 147 to 118 tokens). This suggests that the response pattern changes can be task-dependent: math problems benefit from detailed step-by-step reasoning, whereas tool-use tasks favor shorter, concise action sequences. Such task-dependent property also affects the SFT/RL synergy dynamics, as MATH tasks benefit from imitating an expert’s long and verbose reasoning via SFT, while such patterns can be detrimental to tool-use performance, a distinction further detailed in Appendix D.2 . The result shows that the proposed Chord - ϕ \phi strikes a more nuanced balance: while it also learns to produce more comprehensive mathematical reasoning (2,444 tokens), it generates concise and efficient responses for tool-use tasks (120 tokens). This suggests that the token-wise weighting in Chord - ϕ \phi enables the model to selectively integrate patterns from those of expert data in a task-specific manner. Qualitative analysis shown in Appendix E also confirms the effectiveness of such a flexible design, suggesting that the proposed Chord - ϕ \phi can go beyond simply mimicking the expert, and learn to selectively absorb reasoning patterns from the expert, while exploring its own response strategies .
### 4.3 Analysis on the Effects of μ \mu and ϕ ( ⋅ ) \phi(\cdot)
We provide analysis on the effects of the coefficient μ \mu and the token-wise weighting function ϕ ( ⋅ ) \phi(\cdot) .
Dynamic μ \mu Versus Fixed μ \mu In Figure 9 , we compare the model performance when applying a dynamic schedule for μ \mu (decreasing from 0.9 to 0.05 over the first 200 training steps and keeping unchanged in the following steps) against several fixed schedules in Chord . We observe that applying a fixed μ \mu consistently results in poorer performance compared to dynamic μ \mu . This indicates that naively incorporating off-policy SFT data with a static weight does not effectively serve as a solution for simultaneously learning from off-policy data and on-policy exploration. In fact, it might fail to match Pure RL, which directly encourages an instruction model to follow its own reasoning patterns, highlighting the importance and necessity of controlling the influence of off-policy data.
Besides, while using a smaller value of μ \mu (e.g., 0.02) can mitigate the performance degradation compared to larger values (e.g., 0.1 and 0.5), it does not provide a significant improvement over pure on-policy RL. With a fixed μ \mu , the model is consistently required to accommodate two potentially divergent reasoning patterns, which might pull it in different directions and prevent it from converging to a stable and high-performance state. The decay schedule for μ \mu effectively resolves this conflict by creating a smooth transition from off-policy supervision to on-policy exploration.
We also explore a natural extension to adaptively adjust the loss weight μ \mu based on on-policy rewards. Specifically, the weight is computed as μ = max ( 0 , τ − reward_mean ) \mu=\max(0,\tau-\text{reward\_mean}) , which phases out the SFT loss on the expert data as performance improves. The experiment results in Appendix B.1 imply that an automated, reward-aware schedule for μ \mu can work effectively but requires heavy hyper-parameter tuning, which further motivates the need for fine-grained control.
Training Curve of Chord - ϕ \phi In Figures 9 and 9 , we compare the entropy loss and rewards of Pure RL with those of Chord - ϕ \phi (with fixed μ = 0.1 \mu=0.1 ), to illustrate their training dynamics.