On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting — part 5/5

Cleaned-text code points [63296, 77430). Bibliography omitted here; source.md includes it.

In traditional RL domains like robotics ( Kober et al., 2013 ) or games ( Mnih et al., 2015 ) , combining on-policy and off-policy data is a potent strategy. Methods ranging from alternating training phases ( Gao et al., 2025 ) , to mixing data from separate buffers ( Ball et al., 2023 ) , or directly augmenting on-policy replay buffers with expert trajectories ( Nachum et al., 2017 ) have been proven useful. While such methods yield good results in the traditional RL fields, the discrepancy arises from two fundamental distinctions of LLMs: their strong initial priors, where aggressive off-policy updates risk disrupting established reasoning patterns, and their vast, autoregressive action space that radically increases the off-policy degree of expert data, especially for long reasoning chains, and invalidates the assumptions underpinning conventional off-policy algorithms.

#### Combining On-policy and Off-policy Data in RL for LLM

Leveraging off-policy data to improve the sample efficiency is a well-established strategy in RL. Several studies have focused on leveraging stale, self-generated data by employing techniques such as refining importance sampling corrections ( Tang et al., 2025 ) , mixing on- and off-policy gradients ( Li and Khashabi, 2025 ) , modifying the optimization loss objective ( Roux et al., 2025 ; Arnal et al., 2025 ) , or adjusting the synchronization frequency between online and target policies ( Lanchantin et al., 2025 ) .

More closely related to our work are methods that leverage external expert data to guide the reinforcement learning process for LLMs. These methods can be broadly categorized. One strategy is direct data mixing ( Yan et al., 2025 ; Dong et al., 2025 ; Li and Khashabi, 2025 ) . For example, SimpleMix ( Li and Khashabi, 2025 ) , operates within a DPO framework and combines off-policy and on-policy data via simple dataset-level sampling. LUFFY ( Yan et al., 2025 ) on the other hand, incorporates off-policy expert trajectories directly into the on-policy rollout groups within a GRPO framework. While such approaches expose the model to expert data, they also introduce significant constraints: they usually require strict prompt alignment between datasets or lack the dynamic, token-level weighting needed to manage severe distribution shifts. Another strategy involves using expert data as guidance for generation. For instance, UFT ( Liu et al., 2025a ) and BREAD ( Zhang et al., 2025b ) utilize supervised fine-tuning (SFT) trajectories as prefixes for on-policy rollouts; UFT progressively masks the suffix of the expert demonstration, while BREAD initiates new rollouts by branching from intermediate steps. A third category interleaves RL updates with SFT steps on expert data, either selectively for challenging examples ( Ma et al., 2025 ) or based on a probabilistic schedule ( Chen et al., 2025c ) . Most recently, SRFT ( Fu et al., 2025 ) unifies these approaches into a single-stage framework by not only mixing SFT samples into the on-policy rollout groups but also applying a dedicated SFT loss whose influence is adjusted at the sample level.

Our work diverges from these methods in a crucial aspect. The aforementioned approaches, including state-of-the-art methods like SRFT ( Fu et al., 2025 ) , LUFFY ( Yan et al., 2025 ) , and Reift ( Ma et al., 2025 ) , primarily operate under a “zero-RL” paradigm, initiating training from a base model with a nascent policy. In stark contrast, our work addresses the challenge of fine-tuning a model that already possesses a well-developed, instruction-following policy. This advanced starting point inherently creates a more significant distributional shift between the model’s existing policy and the external expert data, thereby exacerbating the off-policy correction problem that our method aims to solve. For further empirical analysis and results, please refer to Appendix D.1 .

## Appendix D Further Discussions

### D.1 The Influence of Off-Policy Data on Base vs. Instruction Models

The challenges of controlling the influence of off-policy data and maintaining training stability are significantly amplified when fine-tuning instruction models. This is mainly due to the established policy inherent in these instruction models.

Starting from Base Model vs. Instruct Model A base model, having been pre-trained solely with a language modeling objective, lacks a coherent, task-specific policy for instruction following. It often has not yet converged on a particular response pattern. When learning from off-policy expert data, the training process is akin to initial policy formation. The model learns a new skill without the risk of conflicting with an existing pattern, thus avoiding significant instability during training.

In contrast, an instruction model has already developed a sharply-peaked policy. Training these models on off-policy expert data that may reflect different reasoning patterns introduces a substantial distributional mismatch . The RL algorithm’s efforts to reconcile this mismatch can result in large, disruptive policy updates, destabilizing the established policy and potentially leading to a collapse in performance.

Figure 13 provides empirical observation to support the above discussions. When learning from a mixture of on-policy and off-policy data, the reward of a base model improves monotonically, displaying none of the instability issues that can affect instruction models under similar conditions.

Different from most existing studies ( Zeng et al., 2025 ; Yan et al., 2025 ; Fu et al., 2025 ) , which focus on the “Zero-RL” setting that trains from a base model, this paper addresses a more challenging yet practical problem: how to effectively integrate knowledge from off-policy experts into a model that already possesses an established policy. Training from a base model is not always feasible in practical applications. For instance, such methods are ineffective for tool-use tasks, as the base model typically lacks the basic capability to follow the necessary instructions.

Applying “Zero-RL” Methods to Our Setting To demonstrate the unique advantages of Chord for aligning already instruction-tuned models, we conduct additional experiments comparing our proposed Chord with LUFFY ( Yan et al., 2025 ) and SRFT ( Fu et al., 2025 ) on the tool-use task. Note that both LUFFY and SRFT require strict alignment between expert demonstrations and RL prompts, as they directly mix expert trajectories into on-policy rollouts. Hence, we generate expert demonstrations for all 5,000 training prompts using DeepSeek-R1. In contrast, Chord only uses 500 expert demonstrations without requiring prompt-level alignment.

The results in Table 12 , show that Chord significantly outperforms both methods. As discussed in Appendix C , when applied to instruction-tuned models with established policies, directly mixing expert trajectories causes significant distributional mismatch, leading to training instability. Specifically, LUFFY’s upweighting of low-probability tokens on top of importance sampling can still cause policy shifts when the distribution gap between expert and policy gap is large. SRFT’s uniform sample-level weighting cannot distinguish valuable tokens from irrelevant ones within a trajectory, leading to inefficient and misguided updates. In contrast, our ϕ ⁡ ( ⋅ ) \phi(\cdot) function provides token-wise adaptive weighting, enabling selective absorption of expert patterns while maintaining policy stability. These results validate that our method achieves superior performance with better expert data efficiency and maintains training stability on instruction-tuned models, making it more practical for many more real-world applications.

### D.2 On Different Task-Related Performance

The differing performance gains on the MATH and tool-use tasks stem from the fundamental distinctions between these two domains. We deliberately chose these tasks to represent two distinct paradigms, thereby demonstrating the robustness and flexibility of our proposed method.

The math domain benefits from complex, structured, and long-form reasoning. For such tasks, acquiring the necessary problem-solving patterns through pure on-policy exploration (i.e., Pure RL) can be inefficient in comparison. Supervised Fine-Tuning (SFT) on expert data is highly beneficial in this context, as it directly exposes the model to well-structured, step-by-step reasoning chains. This allows the model to efficiently learn complex reasoning frameworks that are difficult to discover from scratch. As we discussed in Section 4.2 , the model’s performance on math problems often correlates with its ability to produce more comprehensive and detailed reasoning steps, a pattern effectively taught by expert data. Therefore, a method that can successfully integrate these expert reasoning patterns, like ours, is expected to yield substantial improvements.

The tool-use domain , in contrast, relies more on the exact tool call result rather than the reasoning process. In this setting, naive imitation of expert trajectories through SFT can even be detrimental, as an expert’s solution may contain stylistic artifacts (e.g., verbosity) that are not conducive to performance. As shown in Table 2 and discussed in Section 4.2 , tool-use tasks favor concise and efficient responses, a pattern that Pure RL naturally learns by shortening response lengths. The primary challenge here is not just to imitate the expert, but to leverage expert guidance to accelerate exploration without being overly constrained or picking up suboptimal habits. The consistent performance gain of our method over the strong Pure RL baseline demonstrates its ability to achieve this delicate balance: successfully extracting useful signals from expert data while avoiding the pitfalls of naive imitation.

These two domains present different challenges for unifying offline SFT and online RL, and our proposed method proves its effectiveness by excelling in both scenarios. It learns to produce comprehensive reasoning for MATH while generating concise, efficient tool calls for tool-use tasks, demonstrating its capability to selectively absorb expert knowledge in a task-specific manner. This validates our approach as a robust and versatile framework for diverse applications.

### D.3 Scaling SFT Is Not Enough: The Necessity of On-Policy Learning

A crucial question is whether extensive SFT on high-quality expert data could eliminate the need for combining SFT and RL. Indeed, as the quantity and diversity of data increase, the problem of exposure bias ( Zhang et al., 2019 ) can be alleviated, leading to better generalization. And for knowledge-intensive tasks like MATH, model performance can be highly correlated with the volume and quality of SFT data. To investigate this, we expanded the MATH SFT dataset from 5k to 20k examples, which substantially boosted the pure SFT model’s AIME accuracy from 15% to approximately 24%.

However, even with larger volumes of SFT data, a principled transition to on-policy learning remains critical for reaching the performance frontier. Recent literature ( Liu et al., 2025b ) also shows that extensive SFT followed by RL fine-tuning is an effective strategy for maximizing model capabilities. By applying our SFT/RL combined approach, we can further elevate the accuracy from 24% to 33%. This demonstrates that RL is not redundant but complementary, enabling the model to refine its policy beyond the static distribution of expert data.

## Appendix E Case Studies

For a better understanding, we compare the generation patterns of RL-only (i.e., pure RL), SFT-only, and the proposed Chord .

• RL-only : The model trained solely with RL exhibits a concise and structured, yet ultimately rigid, reasoning pattern. It tends to follow a fixed template, such as beginning with “To…” and using connectors like “First” and “Next”, and proceeds linearly without engaging in self-correction or exploring alternative solution paths. While this approach leads to efficient responses, it may result in less robust solutions.

• SFT-only and Chord - μ \mu : In contrast, the model trained solely on expert demonstrations can be verbose and exploratory. It exhibits a “think-aloud” style with frequent meta-commentary (e.g., “Let me think…”), backtracking, and verification of intermediate steps. The generated responses are comprehensive, but often lack a concise structure. Note that the proposed Chord - μ \mu has a similar reasoning pattern to SFT-only.

• Chord - ϕ \phi : It exhibits a hybrid reasoning style that retains a clean and logical structure while selectively incorporating the expert’s sophisticated verification strategies. As shown in the example below, it develops patterns such as “Alternatively…” and “Both methods confirm…” to produce responses that are both well-structured and robust. We select the checkpoints at 800 steps as the models to generate the examples here.

## Use of Large Language Models

We used large language models only as general-purpose writing assistants, to proofread and correct grammatical errors in this manuscript.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .