On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting — part 4/5

Cleaned-text code points [47651, 63296). Bibliography omitted here; source.md includes it.

• Entropy Norm : Normalizes the SFT loss weights based on entropy magnitude, with ϕ ⁡ ( p t ) ∝ H ⁡ ( t ) \phi(p_{t})\propto H(t) .

• IS Clip : Applies importance sampling correction but clips tokens with p t > 0.4 p_{t}>0.4 .

• Focal Loss : Adapts focal loss ( Lin et al., 2017 ) to the SFT context, giving higher weight to tokens with lower probability: ϕ ⁡ ( p ) = ( 1 − p ) γ \phi(p)=(1-p)^{\gamma} .

#### Experimental Setup.

For the mathematical reasoning experiments here, we relax the strict on-policy training protocol: we synchronize the policy model every 2 training steps (instead of after each update) and increase the number of rollouts per prompt to 16. Training is conducted for 400 steps. For tool-use tasks, we maintain the same setup as described in Appendix A .

#### Results and Analysis.

Table 4 presents the results on the ToolACE benchmark, while Table 5 shows the performance on mathematical reasoning tasks.

The experimental results reveal several interesting patterns across the different weighting strategies: • Chord - ϕ \phi (Ours): Our proposed method achieves consistently strong performance across both tool-use and mathematical reasoning benchmarks, demonstrating its effectiveness and robustness.

• Entropy-based Variants (Entropy Top & Norm): These methods validate the intuition of focusing on uncertain tokens. Entropy Top shows particularly strong performance on AIME2024 (17.2), proving that selectively emphasizing high-entropy tokens can effectively integrate expert knowledge. However, their performance gains are not as consistent as our method across all benchmarks.

• IS Clip: This variant, which clips high-probability tokens, shows limited effectiveness and even underperforms the pure RL baseline on the tool-use task. This suggests that simply clipping tokens is not a sufficiently nuanced strategy.

• Focal Loss: This strategy, which aggressively up-weights low-probability (high-surprise) tokens, leads to severe training instability and a significant performance collapse on both task types. This confirms our hypothesis that giving excessive weight to tokens the model deems unlikely can disrupt its learned reasoning abilities and lead to overfitting on expert patterns.

These results highlight the importance of fine-grained control in token-wise weighting. While various strategies can provide improvements over pure RL in specific scenarios, the choice of weighting function significantly impacts both training stability and final performance across different task domains. We note that our proposed ϕ ⁡ ( ⋅ ) \phi(\cdot) instantiation represents one effective realization of the general principle of down-weighting tokens at both probability extremes. The varied performance of different variants suggests that there remains room for exploring alternative weighting schemes that may be better suited to specific task characteristics or training scenarios, and we hope these empirical observations can inspire future research in this direction.

### B.3 Varying Expert Data Source

To further validate the robustness and generalizability of our approach, we conduct additional experiments using expert demonstrations generated by Qwen2.5-72B-Instruct instead of DeepSeek-R1. This setup is particularly interesting because Qwen2.5-72B-Instruct, while being a weaker expert model compared to DeepSeek-R1, produces responses with reasoning patterns that are more aligned with the base LLaMA3.2-3B-Instruct model. This allows us to investigate how the choice of expert data source—and specifically, the degree of pattern shift introduced—affects the effectiveness of different training methods.

Table 6 presents the performance comparison on the BFCL benchmark using expert data from both DeepSeek-R1 and Qwen2.5-72B-Instruct. The results lead to several key insights.

When using expert data from Qwen2.5-72B-Instruct, which exhibits reasoning patterns closer to those of LLaMA3.2-3B-Instruct, Chord - μ \mu achieves improved performance (78.1 vs. 77.6) compared to using DeepSeek-R1 data. This validates our hypothesis that the distributional shift introduced by expert data is a critical factor. When the expert’s reasoning pattern is more compatible with the base model’s existing policy, the progressive integration strategy of Chord - μ \mu can more effectively leverage this alignment, leading to better final performance.

Despite the weaker quality of Qwen2.5-72B-Instruct compared to DeepSeek-R1, Chord - ϕ \phi maintains strong and consistent performance (78.3 vs. 78.5) across both expert data sources. This demonstrates the robustness of the token-wise weighting mechanism, which allows the model to selectively absorb useful patterns while mitigating the negative effects of distributional mismatch, regardless of the expert’s absolute strength or stylistic differences.

Interestingly, the baseline SFT-best + RL method also shows notable improvement when using Qwen2.5-72B-Instruct data (77.7 vs. 76.1). This further corroborates our core motivation: the effectiveness of SFT is not solely determined by the quality of expert demonstrations in isolation, but is also heavily influenced by the degree of pattern shift they introduce relative to the base model. A smaller pattern shift, even from a weaker expert, can be more beneficial than a larger shift from a stronger but stylistically divergent expert.

#### Pattern Examples

To provide qualitative insight into these pattern differences, we present example responses from DeepSeek-R1, Qwen2.5-72B-Instruct, and the LLaMA3.2-3B-Instruct model after pure RL training below.

As illustrated in these examples, DeepSeek-R1 produces much more verbose and elaborate reasoning with extensive meta-commentary, while Qwen2.5-72B-Instruct adopts a more concise style that is closer to the direct, structured approach learned by LLaMA3.2-3B-Instruct through pure RL training. This qualitative analysis confirms that Qwen2.5-72B-Instruct introduces a smaller pattern shift, which aligns with the quantitative improvements observed in Table 6 .

### B.4 Non-Verifiable Tasks

To further test the generalizability of Chord beyond verifiable tasks, we conduct additional experiments on the RaR-Medicine dataset ( Gunjal et al., 2025 ) , a medical question-answering task that requires reasoning and explanation without deterministic verification.

We perform training on the Qwen2.5-7B-Instruct model for 200 steps, with both SFT and RL batch sizes of 96, 8 rollouts per prompt, a learning rate of 1 × 10 − 6 1\times 10^{-6} , and the μ \mu decay step is set to 50. We use Qwen3-30B-3A-Instruct as the judge model. The expert demonstrations are sourced from the English subset of the medical-o1-reasoning dataset ( Chen et al., 2025d ) , which contains high-quality reasoning traces for medical questions.

Table 7 and Figure 10 present the experimental results. Both Chord - μ \mu and Chord - ϕ \phi significantly outperform pure RL, achieving testset scores of 80.6 and 81.3 compared to 76.8 for pure RL. Figure 10 further show that Chord - ϕ \phi achieves faster convergence and higher final rewards compared to pure RL, indicating more efficient exploration guided by expert demonstrations, where Chord - μ \mu possesses a similar “shift-readapt” pattern similar to the main experiment. These results validate that our approach can further generalize to more diverse post-training domains.

### B.5 SFT/RL synergy for Weaker Policy Models

An important consideration is how the SFT/RL synergy works when the initial policy model is less capable. Intuitively, one might assume that for a weaker model, supervised fine-tuning (SFT) on expert data would become more critical, as the model’s own on-policy exploration is likely to be less effective.

However, our experiments reveal a more nuanced reality: a weaker model can also struggle to effectively absorb knowledge from expert data. As shown in Table 8 , naively fine-tuning the weaker Qwen2.5-3B-Instruct model with SFT leads to a performance collapse(RL settings are similar to Appendix B.2 ). This is in stark contrast to the result observed when training the Qwen2.5-7B-Instruct model, whose performance significantly improves with the same 5k SFT samples (e.g., AIME2024 accuracy rising from 11.7% to 15.8%). For the weaker model, while Pure RL still provides a consistent performance lift, a naive SFT+RL combination proves unstable. This instability highlights a dual limitation: a weaker model is constrained not only in its on-policy exploration but also in its capacity to absorb off-policy expert data, making the trade-off between exploration and imitation challenging to navigate. In such a setting, the Chord - ϕ \phi method successfully maintains learning stability by providing a robust objective to balance this trade-off. We acknowledge that with larger quantities of higher-quality expert data, a weaker model might overcome this imitation bottleneck, potentially making SFT-leaning methods more favorable. Conversely, this also suggests that our method becomes particularly advantageous when dealing with weaker models under limited expert data, demonstrating its strong capability to flexibly adapt and create a robust synergy between SFT and RL even when naive imitation fails.

### B.6 Diverse Model Architectures

To assess the broader applicability of our method, we extended our evaluation to a model with a distinct architecture and origin: the Phi-mini-MoE-instruct model ( Li et al., 2025 ) (a light-weight Mixture of Experts (MoE) model with 3.8B total, 1.1B active params). This experiment also tests our method’s effectiveness beyond the dense Qwen and LLaMA models.

As shown in Table 9 , the MoE model exhibits a similar vulnerability to naive SFT in tool-use tasks, with performance collapsing significantly. In stark contrast, our Chord - ϕ \phi effectively achieves the highest performance and boosts the overall accuracy from 49.5 to 61.6.

These results show that our method is architecture-agnostic, and further demonstrates its effectiveness across diverse model families and architectures.

### B.7 Tuning μ \mu in Conjunction with ϕ \phi

The proposed Chord employs a dual-control mechanism: a global coefficient μ \mu and a token-wise weighting function ϕ ⁡ ( ⋅ ) \phi(\cdot) . While this raises the question of their joint scheduling, we find that the fine-grained control from ϕ ⁡ ( ⋅ ) \phi(\cdot) makes the framework more robust to the specific schedule of μ \mu . This innovation alleviates the need for meticulous tuning of the global coefficient, simplifying the practical application of Chord .

The aggressive decay schedule for μ \mu (starting from a high value) was designed to manage the “shift-readapt” progression. However, since the weight function ϕ ⁡ ( ⋅ ) \phi(\cdot) also aims to stabilize learning and prevent pattern disruption, such an aggressive start may be unnecessary. A more theoretically aligned approach would be to gently introduce the expert data via a warmup-then-decay ( Hu et al., 2024 ) schedule for μ \mu (e.g., warming up from 0 to 0.3 before decaying). This would align with the stabilizing nature of ϕ ⁡ ( ⋅ ) \phi(\cdot) .

We compare these two schedules in Figure 12 . Although Chord -tune-both that leverages a more refined warmup-then-decay μ \mu schedule yields a slightly better reward progression during training, the final performance gap between the two approaches is not that significant.

This observation is consistent with our insight: the primary purpose of introducing ϕ ⁡ ( ⋅ ) \phi(\cdot) is to enable expert data to continuously and stably guide exploration. By inherently preventing both the disruption of existing patterns and overfitting at a token level, ϕ ⁡ ( ⋅ ) \phi(\cdot) makes the aggressive expert-first approach (a large initial μ \mu ) less critical. The token-wise control provides stability, making the overall system less sensitive to the global trade-off hyperparameter. We argue that adopting ϕ ⁡ ( ⋅ ) \phi(\cdot) not only improves stability but also simplifies the practical application of our framework by making it robust to the specific choice of the μ \mu schedule.

### B.8 Experimental Results on Tool-use Training

We provide the training curves on tool-use tasks in Figure 12 and a more detailed experimental result on the BFCL benchmark in Table 10 . The average performance reported in the BFCL benchmark is averaged by instance, meaning that categories with more instances have a greater contribution to the final average score. All methods are evaluated using the same system prompt format.

### B.9 Experimental Results on the MMLU-pro dataset

We provide a more detailed experimental result on the MMLU-pro dataset in Table 11 . The adopted prompts for generating these results can be found in Appendix A.3 .

## Appendix C Detailed Discussions of Related Works

### C.1 Finetuning for LLMs

#### SFT for LLMs.

SFT has established itself as a cornerstone for aligning LLMs, primarily due to its conceptual simplicity and cost-effectiveness, making it a favored approach within the open-source community for creating capable instruction-following models ( Taori et al., 2023 ; Köpf et al., 2023 ) . Early work emphasized the power of high-quality datasets ( Zhou et al., 2023 ; Young et al., 2024 ) , while the required expert curation is labor-intensive and costly. Moreover, to cover the diverse use cases of modern LLMs, the paradigm has shifted towards massive-scale SFT ( Grattafiori et al., 2024 ; Lambert et al., 2024 ) . This trend makes it computationally prohibitive for many to fine-tune from a base model, promoting continued tuning on pre-aligned instruction models instead. Furthermore, the interplay between SFT and RL has grown more complex, from recent methods like DFT ( Wu et al., 2025 ) or iw-SFT ( Qin and Springenberg, 2025 ) that incorporate RL-inspired importance sampling into SFT, to reasoning models like DeepSeek-R1 ( Guo et al., 2025 ) that strategically integrate both paradigms, highlighting that the optimal, principled integration of these methods remains a critical and open area of research.

#### RL for LLMs.

Recent applications of Reinforcement Learning (RL) for Large Language Models (LLMs) have expanded beyond traditional human preference alignment ( Bai et al., 2022 ; Ouyang et al., 2022 ) , demonstrating significant progress in complex reasoning domains such as mathematics and code generation ( Shao et al., 2024 ; Yang et al., 2024b ; Guo et al., 2025 ) . In particular, a surge of recent work has focused on Reinforcement Learning from Verifiable Rewards (RLVR) ( Lambert et al., 2024 ; Guo et al., 2025 ) , where rewards are derived from definitive outcomes like correct answers or passing unit tests. This paradigm has achieved remarkable results on various benchmarks. However, a fundamental challenge persists in how RL can facilitate effective exploration to surpass the inherent capabilities of its base model ( Yue et al., 2025 ) . The search for novel solutions is often constrained by the model’s pre-existing knowledge, limiting its discovery of superior reasoning pathways. To address this, introducing external expert data — either for distillation ( Hu et al., 2025b ; Liu et al., 2025b ; Guha et al., 2025 ) , cold start ( Guo et al., 2025 ) , or to guide exploration towards diverse, high-quality patterns ( Yan et al., 2025 ; Ma et al., 2025 ) — emerges as a promising approach to transcend these limitations and unlock new problem-solving frontiers.

### C.2 On- and Off-policy Reinforcement Learning

#### Combining On-policy and Off-policy Data in Traditional RL