RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments — part 2/5

Cleaned-text code points [15401, 30680). Bibliography omitted here; source.md includes it.

As a concrete example, many environments in RLVE-Gym are adapted from programming competition problems. In the original problem setting, competitors are asked to write programs that correctly solve the provided test cases. Thus, if the goal were merely to obtain the correct output, one could simply execute such a program on the test case. Instead, the model is required to manually produce the correct outputs without executing code. Through this process, the model could learn reasoning capabilities that can generalize to broader tasks ( Bao et al., 2025 ) . For example, even manually simulating a simple recursive function for brute force could involve problem decomposition, self-verification, and backtracking ( Gandhi et al., 2025 ) , not to mention that successful manual solving often requires more sophisticated reasoning to complete a problem within a reasonable time.

### 3.2 Verification via Environment Advantages

The second principle we used in environment engineering is to verify model outputs by exploiting the advantages of the environment over the LM. In environments constructed under the first principle, writing problem-solving programs is usually easier and more reliable than solving problems manually. As the environment can execute programs while the LM is not allowed to do so, we can exploit this advantage by using such programs (e.g., those originally designed to solve programming problems) for output verification.

Another advantage is that the environment is responsible only for verifying outputs, whereas the LM is tasked with solving the problems. We exploit this advantage when building some environments that exhibit inherent asymmetry between solving and verification in computational complexity, allowing us to eliminate the need for implementing time-consuming solvers altogether. For example, in a Sudoku environment, generation can be performed by masking cells in a randomly sampled complete solution to form a valid puzzle; verification is straightforward using Sudoku rules, whereas solving the puzzle itself requires intractable time complexity, which the environment does not need to implement. NP-complete problems such as SAT ( Cook, 1971 ) or Hamiltonian path detection ( Garey & Johnson, 1979 ; Karp, 1972 ) typically exhibit this asymmetry. Another example is an environment where the model computes the integral of a function: the generator creates an elementary function f f and asks the model to compute the definite integral of f ′ f^{\prime} ; the verifier then simply checks whether the output corresponds to f f , without computing the integral itself. Exploiting this advantage provides supervision signals that are otherwise infeasible to obtain through imitation learning, offering the long-term potential to train LMs on highly complex tasks that humans cannot easily solve themselves.

### 3.3 Designing Configurable Difficulty

We design how incrementing the difficulty level d d leads to harder sampled problems for each environment independently. We achieve this by ensuring that solving any lower-difficulty problem is reducible to, or a subproblem of, solving a higher-difficulty one from the same environment.

For example, in sorting, if a model can sort arrays of length ( N + 1 ) (N+1) , it must also be able to sort any array of length N N , since inserting the smallest element at the beginning of a length- N N array produces a length- ( N + 1 ) (N+1) array whose solution implies the former. Similarly, integrating all functions whose expression trees have ( N + 1 ) (N+1) nodes presupposes the ability to handle all functions with N N nodes (e.g., solving ∫ ( f + 1 ) \int(f+1) implies solving ∫ f \int f ). Thus, in these two environments, a larger difficulty level d d corresponds to a longer array length and a larger expression tree size, respectively.

## 4 Analyzing Components of RLVE

In Sections 4.1 and 4.2 , we study (1) the comparison between adaptive and static environments and (2) the effect of scaling the collection of training environments, respectively.

To facilitate our study, we build our own test set as a controlled evaluation setup. Specifically, we randomly select 50 environments as held-out test environments from RLVE-Gym and sample 50 distinct problems per environment, resulting in a fixed test set 𝒟 ood \mathcal{D}_{\text{ood}} of 2,500 problems in total. From the remaining 350 environments, we randomly select 256 environments to form a collection 𝒞 256 \mathcal{C}_{256} . In all following experiments, every training problem used by RLVE is generated from 𝒞 256 \mathcal{C}_{256} or its subsets, depending on the specific experiment. This setup ensures that the constructed test set 𝒟 ood \mathcal{D}_{\text{ood}} serves as an explicit out-of-distribution (OOD) evaluation that focuses on evaluating generalizable reasoning capabilities. Further details are provided in Appendix D .

We experiment with three types of LMs: (1) base model: Qwen2.5-7B-Base ( Qwen Team, 2024 ) ; (2) SFT model: R1-Distill-Qwen-1.5B ( DeepSeek-AI, 2025 ) ; (3) RL models: DeepScaleR-1.5B ( Luo et al., 2025 ) and ProRL-1.5B-v2 ( Hu et al., 2025a ) .

### 4.1 Adaptivity for Unstalled and Efficient Learning

To examine the effect of adaptive difficulty enabled by RLVE , we compare training on the same verifiable environment with adaptive difficulty against training with static difficulty, where the difficulty distribution remains constant throughout training. We study three types of static difficulty distributions, where problem difficulty is uniformly sampled from d ∼ [ 0 , 1 ] d\sim[0,1] , d ∼ [ 0 , 20 ] d\sim[0,20] , and d ∼ [ 0,100 ] d\sim[0,100] , respectively.

As a case study, we focus on training Qwen2.5-7B-Base (1) on Sorting , where the model is asked to sort a given array, and (2) on Multiplication , where the model is asked to compute the product of two integers, and higher difficulty corresponds to operands with more digits. For evaluation, we measure both in-distribution (ID) performance on a fixed set of held-out 4,000 problems generated from the same training environment with difficulty evenly sampled from [ 0 , 20 ) [0,20) , and OOD performance on 𝒟 ood \mathcal{D}_{\text{ood}} .

From Figure 3 , we observe that when the static environment has a relatively low upper-bound difficulty, the effective prompt ratio eventually drops to zero, indicating that the model masters the hardest problems within the environment’s static difficulty distribution after a certain amount of compute and subsequently becomes saturated. As a result, learning stalls once the environment ceases to provide learning signals. Prior works that train LMs on data generated by static verifiable environments have similarly been observed to suffer from early saturation ( Li et al., 2025 ) .

When the static environment instead has a high upper-bound difficulty such that the model cannot master all problems within a limited compute budget, the effective prompt ratio remains nonzero. However, the ratio drops substantially below that of adaptive difficulty, indicating only a small fraction of problems from the static environment are appropriately challenging. Empirically, this discrepancy significantly impairs both ID and OOD performance.

It is worth noting that training with the static difficulty distribution d ∼ [ 0 , 20 ] d\sim[0,20] confers an oracle advantage, as its difficulty distribution coincides with that of the ID evaluation. In realistic scenarios, finding such an oracle is infeasible. Even without such an oracle advantage, RLVE achieves comparable or even superior ID performance.

In conclusion, adaptive difficulty both (1) prevents learning from stalling due to overly easy problems and (2) avoids learning inefficiency caused by a large proportion of problems that are inappropriately challenging for the policy.

One might argue that an optimal static difficulty distribution could be manually tuned for a given compute budget. However, this manual tuning becomes infeasible when training jointly across many verifiable environments. To illustrate this, we train Qwen2.5-7B-Base , R1-Distill-Qwen-1.5B , DeepScaleR-1.5B , and ProRL-1.5B-v2 using RLVE jointly across 𝒞 256 \mathcal{C}_{256} . Figure 4 (a) shows the distribution of upper-bound difficulty levels reached by adaptive environments at step 400, revealing a broad range from 0 to 12. Using this distribution as oracle information, which would not even be available beforehand without adaptive difficulty, we train the same models on all environments from 𝒞 256 \mathcal{C}_{256} using a static difficulty range of [ 0 , 20 ] [0,20] . As shown in Figure 4 (b), although this static range covers every adaptive environment’s difficulty distribution, training with such static environments is consistently outperformed by RLVE . These results suggest that the difficulty of each environment must be individually tuned to match the policy, as each environment defines its own notion of problem difficulty level; such a tuning arises naturally with adaptive environments enabled by RLVE .

### 4.2 Environment Scaling as a Key Driver of Generalizable Reasoning Capabilities

We investigate the effect of scaling up the collection of verifiable environments used during training. Within 𝒞 256 \mathcal{C}_{256} , we construct three fixed environment collections: 𝒞 1 \mathcal{C}_{1} , 𝒞 4 \mathcal{C}_{4} , and 𝒞 16 \mathcal{C}_{16} , containing 1, 4, and 16 distinct verifiable environments, respectively; each larger collection contains all smaller ones, i.e., 𝒞 1 ⊂ 𝒞 4 ⊂ 𝒞 16 ⊂ 𝒞 256 \mathcal{C}_{1}\subset\mathcal{C}_{4}\subset\mathcal{C}_{16}\subset\mathcal{C}_{256} . We train the same four models as in Section 4.1 separately on each environment collection under an identical setup, and evaluate checkpoints on 𝒟 ood \mathcal{D}_{\text{ood}} , which is constructed from the 50 held-out environments. Additional details are provided in Appendix E .

As shown in Figure 5 , expanding the collection of training environments consistently leads to better performance on held-out environments across all model types. From another perspective, merely increasing the volume of RL training data remains insufficient for improving generalizable reasoning capabilities, given that a single environment can already generate an unbounded amount of data. Instead, scaling along the environment dimension emerges as a critical direction for future LM RL training. This insight echoes findings from classical RL research ( Cobbe et al., 2020 ) , and also resonates with observations from other LM training stages, such as SFT ( Wang et al., 2022 ) and embedding learning ( Su et al., 2023 ) , where expanding the collection of tasks matters more than increasing the sheer volume of data.

## 5 Scaling Up RL Training with RLVE

After analyzing components of RLVE , we evaluate RLVE with joint training on all 400 verifiable environments from RLVE-Gym in two representative scenarios for scaling up RL training of LMs: (1) a data-saturation scenario, where the model has already saturated on a strong RLVR dataset (Section 5.1 ); and (2) a compute-constrained scenario, where training starts from a model without prior reasoning RL and the choice of RL data is crucial (Section 5.2 ).

In experiments for these two scenarios, we evaluate models on six representative reasoning benchmarks covering mathematics (AIME 2024/2025 ( Art of Problem Solving, 2025 ) , OMEGA-500 ( Sun et al., 2025 ) , and OlympiadBench ( He et al., 2024 ) ), code generation (LiveCodeBench ( Jain et al., 2025 ) ), and logical reasoning (BBEH ( Kazemi et al., 2025 ) ). Further evaluation details are provided in Appendix D .

### 5.1 Scaling beyond Data Saturation

We first evaluate to what extent RLVE enables further improvement beyond a model that represents one of the best open-sourced efforts to train an LM with RLVR at the time of our study. Specifically, we start from the checkpoint ProRL-1.5B-v2 ( Hu et al., 2025a ) . This starting checkpoint was originally trained from R1-Distill-Qwen-1.5B ( DeepSeek-AI, 2025 ) using over 20,000 H100 GPU hours of RLVR, reaching performance saturation ( Hu et al., 2025b ) on its large and diverse training dataset of approximately 136,000 problems spanning mathematics, coding, logical reasoning, STEM, and instruction-following domains ( Liu et al., 2025e ) ; we refer to this dataset as the ProRL dataset for simplicity.

As a comparison, we also evaluate the checkpoint obtained by continuing the original RLVR training on the ProRL dataset, also initialized from ProRL-1.5B-v2 . As shown in Figures 6 and 8 , within approximately 1,100 H100 GPU hours, RLVE improves the average performance across the six reasoning benchmarks by an absolute 3.37%; in contrast, continuing the original RL training on the ProRL dataset yields only a marginal absolute improvement of 0.49% (7 × \times smaller), even after using more than three times the compute (3,600 H100 GPU hours). Therefore, RLVE effectively scales up RL training beyond the data-saturation point of a model from one of the strongest open-sourced RLVR efforts to date. Importantly, the verifiable environments in RLVE-Gym are designed as pedagogical tools for developing generalizable reasoning capabilities rather than to resemble real LM tasks, and the resulting performance demonstrates effective transfer to real-world reasoning benchmarks.

### 5.2 Compute-Efficient Scaling

We next evaluate to what extent RLVE improves model performance in a compute-constrained scenario, where the available compute budget is fixed and the choice of RL data plays a crucial role. Specifically, we start from OpenThinker3-1.5B ( Guha et al., 2025 ) , which is currently the strongest open-sourced SFT model at the 1.5B scale and has not undergone any reasoning RL training. For comparison, we also train the LM on DeepMath-103K ( He et al., 2025 ) under an identical training setup. DeepMath-103K is an RLVR dataset consisting of approximately 103K high-quality mathematical reasoning problems that are generally more challenging than those in comparable reasoning RLVR datasets, e.g., Luo et al. (2025) ; Yu et al. (2025) .

As shown in Figures 7 and 9 , training with RLVE consistently outperforms training on this existing high-quality RLVR dataset for the same number of steps. The model trained with RLVE consistently achieves higher performance on non-mathematical benchmarks (LiveCodeBench and BBEH) and most mathematical benchmarks (OMEGA-500 and OlympiadBench); RLVE also achieves comparable results on AIME 2024/2025, with peak performance exceeding DeepMath-103K by roughly one point. Thus, despite not targeting any specific benchmark domains, RLVE can significantly foster generalizable reasoning capabilities. Given that constructing DeepMath-103K required about $138,000 USD and 127,000 GPU hours ( He et al., 2025 ) , we offer a far more cost-efficient way to achieve stronger performance. Practically, these experiments also simulate the standard LM post-training pipeline, where LMs are first trained via SFT and subsequently via RL ( DeepSeek-AI, 2025 ; Kimi Team, 2025a ; Qwen Team, 2025 ; GLM-4.5 Team, 2025 ) .

## 6 Related Work

#### RL training of LMs on procedurally generated data.