On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting — part 1/5

Cleaned-text code points [0, 15869). Bibliography omitted here; source.md includes it.

# On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting Thanks: Corresponding author. Email to {zwh434786, yaliang.li}@alibaba-inc.com

###### Abstract

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two prominent post-training paradigms for refining the capabilities and aligning the behavior of Large Language Models (LLMs). Existing approaches that integrate SFT and RL often face the risk of disrupting established response patterns and inducing overfitting to expert data. To address this, we present a novel investigation into the unified view of SFT and RL through an off-policy versus on-policy lens. We propose Chord , a framework for C ontrollable H armonization of O n- and Off-Policy R einforcement Learning via D ynamic Weighting, which reframes SFT not as a separate stage but as a dynamically weighted auxiliary objective within the on-policy RL process. Based on an analysis of off-policy expert data’s influence at both holistic and granular levels, we incorporate a dual-control mechanism in Chord . Specifically, the framework first employs a global coefficient to holistically guide the transition from off-policy imitation to on-policy exploration, and then applies a token-wise weighting function that enables granular learning from the expert, which promotes on-policy exploration and mitigates disruption from off-policy data. We conduct extensive experiments across various practical tasks, providing empirical evidence that Chord achieves a stable and efficient learning process. By effectively harmonizing off-policy expert data with on-policy exploration, Chord demonstrates significant improvements over baselines. We release the implementation to inspire further research.

## 1 Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in a wide array of applications ( Yang et al., 2024b ; Zhang et al., 2025a ; Mialon et al., 2023 ; Gao et al., 2024 ) . Such significant progress can be largely attributed to two critical post-tuning paradigms that enhance the performance of LLMs in real-world scenarios, i.e., Supervised Fine-Tuning (SFT) ( Taori et al., 2023 ; Zhou et al., 2023 ) and Reinforcement Learning (RL) ( Ouyang et al., 2022 ; Shao et al., 2024 ) .

These two paradigms present their pros and cons. SFT relies on high-quality expert trajectories to effectively mimic response patterns, which can be sensitive to the quality and quantity of expert data ( Ye et al., 2025 ; Guha et al., 2025 ) . Recent studies also point out that SFT may struggle to generalize beyond mere memorization ( Chu et al., 2025 ) and is vulnerable to exposure bias ( Zhang et al., 2019 ) . In contrast, RL encourages LLMs to actively explore, which enables better generalization through learning from direct feedback on their on-policy generations ( Chu et al., 2025 ; Chen et al., 2025b ) . However, such explorations can sometimes be inefficient, leading to policy degradation caused by entropy collapse ( Yu et al., 2025 ) or over-exploitation of suboptimal strategies.

A prevalent and straightforward approach for integrating the strengths of SFT and RL while mitigating their weaknesses is the sequential SFT-then-RL paradigm ( Liu et al., 2025b ; Lambert et al., 2024 ) . Intuitively, the expert’s reasoning patterns learned in SFT guide the RL exploration beyond local optima, and then the on-policy learning in RL mitigates exposure bias inherent in SFT and prevents overfitting to a limited set of static examples. However, empirical observations show that the SFT-then-RL paradigm does not consistently outperform the pure RL approach, as illustrated in Figure 2 , which is also noted in recent studies ( Zhang et al., 2025a ; Chen et al., 2025b ) .

In this study, we make a further investigation and demonstrate that such suboptimal performance may arise from training on expert data that significantly diverges from the model’s established patterns. As illustrated in Figure 2 , the learning curve reveals a “shift-readapt-overfit” progression consisting of three distinct phases. Firstly, there is an initial disruption in capability due to the sudden policy shift, which is followed by a readaptation phase during which the model adapts to the expert’s patterns and recovers performance. Finally, we observe that the model eventually overfits the expert data. These observations highlight that while expert data can bring new capabilities, it may also disrupt established patterns and induce overfitting during the training process .

Drawing upon these insights, we unify SFT and RL through the lens of off-policy versus on-policy learning. The SFT process is reframed not as a separate tuning stage, but as a dynamically weighted auxiliary objective within the on-policy RL process. We further design Chord , a framework for C ontrollable H armonization of O n- and Off-Policy R einforcement Learning via D ynamic Weighting. Chord features a global coefficient μ \mu for controlling the overall influence of expert data throughout the training process, and a fine-grained weighting function ϕ ⁡ ( ⋅ ) \phi(\cdot) that helps maintain stability via down-weighting highly divergent tokens from off-policy data that could disrupt on-policy training.

Our contributions can be summarized as follows:

• We provide a systematic and in-depth analysis of the training dynamics when employing a separate SFT process to integrate off-policy expert knowledge into models with established policies. We identify the “shift-readapt-overfit” progression, revealing how off-policy data can disrupt the established response patterns of LLMs.

• We propose Chord , a novel framework that unifies SFT and RL via a dynamically weighted auxiliary loss, which consists of a global coefficient μ \mu and a token-wise weighting function ϕ ⁡ ( ⋅ ) \phi(\cdot) . Chord provides a fine-grained and flexible control of the influence of off-policy expert data while ensuring training stability, promoting a harmonious integration of learning from both off-policy expert demonstrations and the model’s on-policy exploration.

• Extensive experiments demonstrate that Chord outperforms the SFT-then-RL paradigm and existing approaches. We provide both quantitative and qualitative analyses to show that Chord strategically navigates training dynamics to selectively absorb expert knowledge without stifling the model’s reasoning capabilities, highlighting its superiority and effectiveness.

## 2 Preliminaries

The post-tuning of Large Language Models (LLMs) involves optimizing their policy, denoted by π θ \pi_{\theta} and parameterized by θ \theta , to generate desirable responses. This typically follows two paradigms: Supervised Fine-Tuning (SFT), an off-policy paradigm driven by a static dataset of expert demonstrations; and Reinforcement Learning (RL), an on-policy paradigm guided by dynamic feedback.

Specifically, SFT adjusts the policy π θ \pi_{\theta} to mimic a high-quality, static dataset of N N expert demonstrations, 𝒟 SFT = { ( x i , y i ∗ ) } i = 1 N \mathcal{D}_{\text{SFT}}=\{(x_{i},y^{*}_{i})\}_{i=1}^{N} . Here, x i x_{i} is a prompt and y i ∗ = ( y i , 1 ∗ , … , y i , | y i ∗ | ∗ ) y^{*}_{i}=(y^{*}_{i,1},\dots,y^{*}_{i,|y_{i}^{*}|}) is the corresponding expert response with | y i ∗ | |y_{i}^{*}| tokens. The SFT objective is to minimize the negative log-likelihood of expert responses, typically optimized with an empirical estimate from a mini-batch of size B B : ℒ SFT ( θ ) = − 1 ∑ i = 1 B | y i ∗ | ∑ i = 1 B ∑ t = 1 | y i ∗ | log π θ ( y i , t ∗ | x i , y i , < t ∗ ) . \mathcal{L}_{\text{SFT}}(\theta)=-\frac{1}{\sum_{i=1}^{B}|y_{i}^{*}|}\sum_{i=1}^{B}\sum_{t=1}^{|y_{i}^{*}|}\log\pi_{\theta}(y^{*}_{i,t}|x_{i},y^{*}_{i,<t}). (1)

In contrast, RL optimizes policy π θ \pi_{\theta} by maximizing expected reward R ⁡ ( τ ) R(\tau) from a generated trajectory τ = ( x , y ∗ ) \tau=(x,y^{*}) . Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) suggests sampling K K responses { τ 1 , … , τ K } \{\tau_{1},\dots,\tau_{K}\} from a policy π sample \pi_{\text{sample}} when given a prompt x x . Each response τ k \tau_{k} is evaluated with the reward function R ⁡ ( τ k ) R(\tau_{k}) , and π θ \pi_{\theta} is updated to maximize a PPO-style clipped surrogate objective. Consistent with recent studies ( Hu et al., 2025a ; Yu et al., 2025 ; Chen et al., 2025a ) , our formulation does not include the KL divergence term to avoid restricting performance of LLMs. The objective function can be formulated as: ℒ GRPO ( θ ) = − 1 ∑ i = 1 B ^ ∑ k = 1 K | τ i , k | ∑ i = 1 B ^ ∑ k = 1 K ∑ t = 1 | τ i , k | min ( r i , k , t ( θ ) A i , k , clip ( r i , k , t ( θ ) , 1 − ϵ , 1 + ϵ ) A i , k ) , \mathcal{L}_{\text{GRPO}}(\theta)=-\frac{1}{\sum_{i=1}^{\hat{B}}\sum_{k=1}^{K}|\tau_{i,k}|}\sum_{i=1}^{\hat{B}}\sum_{k=1}^{K}\sum_{t=1}^{|\tau_{i,k}|}\min\left(r_{i,k,t}(\theta)A_{i,k},\,\text{clip}(r_{i,k,t}(\theta),1-\epsilon,1+\epsilon)A_{i,k}\right), (2)

where B ^ \hat{B} is the number of prompts in the mini-batch and ϵ \epsilon is the clipping hyper-parameter. The advantage A k A_{k} for each response is computed by A k = R ⁡ ( τ k ) − μ ℛ σ ℛ + ϵ z A_{k}=\frac{R(\tau_{k})-\mu_{\mathcal{R}}}{\sigma_{\mathcal{R}}+\epsilon_{z}} , where μ ℛ \mu_{\mathcal{R}} and σ ℛ \sigma_{\mathcal{R}} are the mean and standard deviation of rewards { R ⁡ ( τ k ) } k = 1 K \{R(\tau_{k})\}_{k=1}^{K} within the group, and ϵ z \epsilon_{z} is a small constant for stability. Here r i , k , t ​ ( θ ) ≜ π θ ​ ( τ i , k , t | x , τ i , k , < t ) π sample ​ ( τ i , k , t | x , τ i , k , < t ) r_{i,k,t}(\theta)\triangleq\frac{\pi_{\theta}(\tau_{i,k,t}|x,\tau_{i,k,<t})}{\pi_{\text{sample}}(\tau_{i,k,t}|x,\tau_{i,k,<t})} denotes the token-wise Importance Sampling (IS) ratio, which re-weights the probability of actions sampled under π sample \pi_{\text{sample}} to simulate on-policy sampled distribution. For a “strict on-policy setup” ( Liu et al., 2025b ) that π sample = π θ \pi_{\text{sample}}=\pi_{\theta} , this ratio should always be 1 1 , and the gradient of r i , k , t ​ ( θ ) r_{i,k,t}(\theta) should be equivalent to ∇ θ ​ log ​ π θ ​ ( τ i , k , t ∗ | x i , τ i , k , < t ∗ ) \nabla_{\theta}\log\pi_{\theta}(\tau^{*}_{i,k,t}|x_{i},\tau^{*}_{i,k,<t}) .

## 3 Chord : Harmonizing Off-Policy and On-Policy Learning

### 3.1 The Shift-Readapt-Overfit Progression When Utilizing Off-Policy Data

Before introducing Chord , we first take a close look at the training dynamics of the SFT process, revealing how training on off-policy expert data can disrupt the established response patterns of LLMs. Such disruption ultimately leads to the failure of the SFT-then-RL paradigm ( Zhang et al., 2025a ; Chen et al., 2025b ) , as evidenced by the results in Figure 2 .

We train Qwen2.5-7B-Instruct ( Yang et al., 2024a ) on expert data generated by Deepseek-R1 ( Guo et al., 2025 ) and monitor the changes in test accuracy on the MATH-500 dataset. From the experimental results shown in Figure 2 , we observe that model performance declines during the first few epochs, followed by a continuous increase to a level higher than that before training, and then a slight subsequent decrease. The performance curve reveals a “shift-readapt-overfit” progression: • Policy Shift : The performance initially declines since the model is forced to follow off-policy expert demonstrations whose response patterns are significantly different, disrupting its established response patterns and causing a significant performance drop . Such degradation is further exacerbated by exposure bias ( Zhang et al., 2019 ; Schmidt, 2019 ) , as the model, trained exclusively on ground-truth expert data, struggles to navigate the self-generated contexts it encounters during inference.

• Readapt : As SFT continues, the model policy π θ \pi_{\theta} begins to integrate the expert’s response patterns and generates responses similar to those of the expert. The exposure bias can be mitigated by reducing the reliance on the model’s response patterns, thereby allowing its performance to rise steadily as it adapts to the expert’s response patterns.

• Overfit : Extended training on the limited expert data ultimately leads to overfitting, resulting in a decline in generalization and a significant loss of output diversity. Such overfitting can also restrict the exploratory capacity that is crucial for the following RL optimization.

The observed progression makes it challenging to control the influence of off-policy expert data. The SFT-then-RL paradigm demands careful timing for the SFT-to-RL transition, and even then, such a two-stage paradigm may still yield suboptimal solutions due to the inherent separation of the training phases. This highlights the limitations and fragility of the SFT-then-RL paradigm, especially when expert data’s response patterns significantly diverge from the model’s established response patterns.

Drawing upon the above insights, we propose Chord , a novel framework that effectively unifies SFT and RL. The proposed framework consists of a dual-control mechanism. We first introduce a dynamic loss coefficient to balance learning from on- and off-policy data (refer to Section 3.2 ), then further design a token-wise weighting function that provides fine-grained stability control (refer to Section 3.3 ). The overall architecture of Chord is shown in Figure 3 .

### 3.2 Controlling the Influence of Off-Policy Expert Data via μ \mu

Firstly, in order to control the influence of off-policy expert data, we propose to reframe SFT as a dynamically weighted auxiliary objective within the on-policy RL process, rather than a separate tuning stage as in the SFT-then-RL paradigm. Specifically, we design a combined loss function that minimizes a weighted sum of the RL and SFT losses: ℒ Hybrid ​ ( θ ) = ( 1 − μ ) ​ ℒ GRPO ​ ( θ ) + μ ​ ℒ SFT ​ ( θ ) , \mathcal{L}_{\text{Hybrid}}(\theta)=(1-\mu)\mathcal{L}_{\text{GRPO}}(\theta)+\mu\mathcal{L}_{\text{SFT}}(\theta), (3) where ℒ GRPO ​ ( θ ) \mathcal{L}_{\text{GRPO}}(\theta) is the empirical GRPO loss defined in equation 2 , ℒ SFT ​ ( θ ) \mathcal{L}_{\text{SFT}}(\theta) is the SFT loss defined in equation 1 , and μ ∈ [ 0 , 1 ] \mu\in[0,1] is a hyperparameter that governs the trade-off between SFT and RL.

If using a fixed value of μ \mu , the influence of the off-policy expert data remains unchanged throughout the entire post-tuning process. An advanced strategy, however, is to change μ \mu for achieving a dynamic balance between off-policy and on-policy learning. For example, the SFT-then-RL pipeline can be regarded as a special case with a binary schedule (initially setting μ = 1 \mu=1 and then transitioning to μ = 0 \mu=0 ). Moreover, previous studies ( Ma et al., 2025 ; Gao et al., 2025 ) that utilize interleaved SFT and RL can be interpreted as employing a periodic μ \mu schedule.

Moving a step forward, applying a decay schedule of μ \mu provides a more graceful and flexible transition from off-policy imitation to on-policy optimization compared to the rigid and binary switch. As shown in Figure 5 , the training begins with a large μ \mu value, encouraging the model to learn more from off-policy expert data. As training progresses, μ \mu gradually decays to a smaller value, shifting the training focus towards on-policy exploration and annealing the influence of the off-policy expert data before overfitting on them. Such a decay schedule has also proven successful in mitigating exposure bias ( Zhang et al., 2019 ) . Inspired by scheduled sampling ( Bengio et al., 2015 ) , we generalize the principle of mixing expert and self-generated data from the token level to the loss formulation, effectively bridging the distributional gap between training on off-policy samples and performing on-policy rollouts.