It Takes Two: Your GRPO Is Secretly DPO — part 5/5
Cleaned-text code points [61698, 69080). Bibliography omitted here; source.md includes it.
2. The probability of obtaining at least one correct answer when performing m m consecutive trials of 2 2 independent rollouts each, with the corresponding policy [ π 0 , π 1 , ⋯ , π m − 1 ] [\pi_{0},\pi_{1},\cdots,\pi_{m-1}] is P m × 2 = 1 − ∏ i = 0 , ⋯ m − 1 ( 1 − p i ) 2 ≥ 1 − ( 1 − p 0 ) 2 m = P 2 m P_{m\times 2}=1-\prod_{i=0,\cdots m-1}(1-p_{i})^{2}\geq 1-(1-p_{0})^{2m}=P_{2m} (31) when we have p i ≥ p 0 , ∀ i > 0 p_{i}\geq p_{0},\forall i>0 .
Note that the assumption p i ≥ p 0 , ∀ i > 0 p_{i}\geq p_{0},\forall i>0 is prevailing, as we assume that the reasoning ability of LLM can be improved by RL post-training.
Proposition C.4 suggests that for difficult questions, 2-GRPO does not degrade in effectiveness compared to 16-GRPO given the same budget of the total number of rollouts in whole training process. Notably, due to its higher frequency of policy updates, 2-GRPO may yield a higher probability of generating correct outputs for hard questions. It is also more adaptive, allowing it to capture nuanced update requirements for varying inputs. This observation also extends to PPO with the standard single-rollout implementations per epoch against multi-rollout variants.
## Appendix D Experiments
### D.1 Experiment Details
#### Dataset and Baselines
For math reasoning task, following prior work [ 6 ] , we employ Qwen2.5-Math-1.5B (Qwen-1.5B) and Qwen2.5-Math-7B (Qwen-7B) [ 41 ] as base models. Both models are post-trained via RL on the MATH [ 14 ] and DAPO-Math-17k [ 42 ] datasets, and evaluated on MATH-500 [ 14 ] , AMC23, Minerva Math [ 18 ] , AIME-2025, and OlympiadBench [ 11 ] . For DAPO-Math-17k dataset, we randomly sample 7.5k questions from the original data to form a subset for training in order to align with the size of MATH. In addition, we assess the proposed method on DeepSeek-R1-Distill-Qwen-1.5B (DS-1.5B) [ 7 ] , which is post-trained on MATH. Owing to computational constraints, we do not extend its post-training to DAPO-Math-17k. All 1.5B models are trained on 4 GPUs with 140GB Memory. Qwen-7B is trained on 8 GPUs with 140GB Memory. We evaluate model performance using two metrics: Mean@32, the average accuracy across 32 i.i.d. samples, and Pass@32, which measures whether a problem is solved in at least one of those 32 attempts.
For visual reasoning task, we use EasyR1 [ 48 ] framework, Qwen2.5-7B [ 2 ] as the base model, and Geometric3K [ 23 ] as the dataset. For code generation task, we use Code-R1 [ 21 ] framework, Qwen2.5-7B-Instruct-1M as the base model, and code-r1-12k 5 5 5 https://huggingface.co/datasets/ganler/code-r1-12k as the dataset. Both visual reasoning and code generation tasks are conducted on 8 GPU.
#### Hyper-parameters
We mainly follow the default configuration of the verl framework. For sampling parameters in training generation, we set temperature to 1, top-p to 1 to encourage exploration, sequence length to 4096 for Qwen-series model and 8192 for DS-1.5B. For sampling parameters in test generation, we set temperature to 0.7, top-p to 0.8, top-k to 20 and sequence length to 4096 for all models. For optimization, training employs the Adam optimizer [ 17 ] with a constant learning rate and a linear warm-up over the first 10 steps. For GRPO hyper-parameters, we set the clip ratio high to 0.28 0.28 and clip ratio lower to 0.2 0.2 following DAPO [ 42 ] . All models are trained for 10 epochs. The baseline method, 16-GRPO, is trained with batch sizes of 32 (32 prompts and 16 rollouts per prompt) and a learning rate 1 × 10 − 6 1\times 10^{-6} . As discussed in Appx. C.2 , we trained 2-GRPO with a larger batch size of 256 (256 prompts and 2 rollouts per prompt). Both case will have 512 rollouts in each mini-batch of training. Since we have fewer update steps due to the larger batch size, we adjust the learning rate of 2-GRPO to 8 × 10 − 6 8\times 10^{-6} based on the linear relationship of learning rate and batch size [ 10 ] .
### D.2 The Connection Between Training Rollouts and Computational Cost
In Sec. 5.1 , the total number of rollouts generated and utilized during training is adopted as a metric for comparing the computational cost of different methods.
The rationale for this choice is as follows. A principled measure of computational cost in the context of RL post-training is the number of floating-point operations (FLOPs) performed. Unlike wall-clock time, which is susceptible to variations arising from software implementation details (e.g., optimization of training libraries) and hardware characteristics (e.g., GPU/CPU architecture, I/O throughput), FLOPs provide a more direct and stable measure of computational effort.
For a fixed base model and the same type of RL algorithm (GRPO in our case), the FLOPs required for a single forward or backward pass with one input prompt can be considered constant, for both the generation and training phases. Accordingly, the total number of rollouts executed during training is directly proportional to the FLOPs executed, thereby serving as a theoretically justified and consistent proxy for computational cost.
### D.3 Sample Discard Rate of 2-GRPO
As discussed in Sec. 4.3 , 2-GRPO may suffer from a high discard rate when prompts are either extremely easy or extremely difficult for the LLM.
In the RLVR setting, the average discard rate P discard P_{\text{discard}} can be estimated from the average reward r ¯ \bar{r} as P discard = 1 − r ¯ 2 − ( 1 − r ¯ ) 2 . P_{\text{discard}}=1-\bar{r}^{2}-(1-\bar{r})^{2}.
To quantify this effect, we report the average discard rate of 2-GRPO using Qwen-7B post-trained on MATH as a representative case (shown in Table 3 ).
### D.4 Ablation Study on Group Size
We present a comprehensive ablation study on group size in Table 4 . To further illustrate this effect, Figure 5 reports the reward curves during training alongside the corresponding validation scores throughout the training process on MATH dataset.
## Appendix E Limitation
The contrastive learning nature of GRPO applies regardless of whether rewards are continuous or binary. However, the present study focuses primarily on the reasoning tasks with the RLVR setting, and we leave the empirical investigation of continuous rewards to future work due to the space limit.
## Appendix F The Use of Large Language Models (LLMs)
We used LLMs in writing, editing and formatting purposes. Our experiments also involve the LLMs.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .