RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments — part 1/5

Cleaned-text code points [0, 15401). Bibliography omitted here; source.md includes it.

# RLVE : Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments

###### Abstract

We introduce R einforcement L earning (RL) with Adaptive V erifiable E nvironments ( RLVE ), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model’s capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE , we create RLVE-Gym , a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym , we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM’s original RL training yields only a 0.49% average absolute gain despite using over 3 × \times more compute. We release our code publicly. 1 1 1 https://github.com/Zhiyuan-Zeng/RLVE

###### Keywords:

## 1 Introduction

Scaling up reinforcement learning (RL) has shown strong potential to improve language models (LMs) ( Ouyang et al., 2022 ; OpenAI, 2024 ; DeepSeek-AI, 2025 ; Google DeepMind, 2025 ) , but models’ improvement increasingly saturates on finite training data ( Kumar et al., 2024 ; Hu et al., 2025b ; Khatri et al., 2025 ) . Scaling up RL data presents two challenges. First, collecting a large number of problems along with their ground-truth answers, which are commonly required for verifiable reward computations ( Lambert et al., 2025 ; DeepSeek-AI, 2025 ) , can be expensive. Second, RL training can completely stall when problems are either too easy or too difficult for the policy model ( Razin et al., 2024 ; Razin et al., 2025 ) , since too easy ones provide no meaningful learning signal, whereas overly difficult ones yield consistently poor rewards that impede gradient-based updates. In typical LM RL training, the problem distribution is predetermined by a specific dataset and remains static, preventing adaptation to the policy model’s evolving capabilities (Figure 1 (a)).

To address these challenges, we introduce RLVE (Reinforcement Learning with Adaptive Verifiable Environments), an approach that scales up LM RL training by using verifiable environments whose difficulty dynamically adapts to the policy model’s capabilities (Figure 1 (b)). A verifiable environment (1) procedurally generates an unbounded number of problems with configurable difficulty, and (2) provides algorithmically verifiable rewards to model outputs. For example, an environment for an array-sorting task can sample an array and verify an output by comparing it with the result produced by a sorting program, overcoming the non-scalability of collecting problems individually; we can increase the difficulty by increasing the array length.

RLVE enables adaptive verifiable environments, in which the difficulty distribution shifts toward harder problems once the policy model performs well on the current distribution. RLVE thus addresses the limitations of static environments, whose distribution of generated problems remains constant throughout training. Empirically, after the model masters the hardest problems, the static environment becomes uninformative for further improvement; even if the model does not ultimately achieve this within a limited compute budget, learning efficiency remains suboptimal when the difficulty of most problems is inappropriate for the policy model. By contrast, our experiments in Section 4.1 show that adaptive difficulty continuously maintains a high proportion of appropriately challenging problems, leading to superior performance and higher learning efficiency.

To implement RLVE , we construct RLVE-Gym , a large-scale suite of 400 verifiable environments developed through our expert environment engineering efforts, following two principles. First, the environments are designed as pedagogical tools for developing reasoning capabilities. This procedure is analogous to teaching a pupil to perform integer multiplication by hand, although using a calculator is more efficient and reliable. Second, the environments enable output verification via two advantages: (1) the environment can execute programs while the LM is not allowed to do so, and (2) the environment is responsible only for verifying outputs rather than solving the problems, and verification is sometimes much easier than solving in terms of computational complexity, e.g., in NP-complete problems. Using RLVE-Gym , Section 4.2 shows that environment scaling , i.e., expanding the training collection of verifiable environments, consistently improves the model performance on environments unseen during training, underscoring its importance for developing generalizable reasoning capabilities.

We show that RLVE with joint training across all 400 environments in RLVE-Gym effectively scales up LM RL training in two scenarios, as reflected by performance on six reasoning benchmarks covering mathematics, code generation, and logical reasoning. The first is a data-saturation scenario. Section 5.1 demonstrates that RLVE further scales up RL training for one of the current strongest 1.5B RL LMs, ProRL-1.5B-v2 ( Hu et al., 2025a ) (Figure 1 (c)). This LM was originally trained with RLVR (RL with verifiable rewards) for over 20,000 H100 GPU hours to saturation on ProRL, a large-scale, diverse dataset ( Liu et al., 2025e ) .

The second is a compute-constrained scenario, where training starts from an LM that has not undergone reasoning RL. Section 5.2 shows that RLVE outperforms training on a strong RLVR dataset, DeepMath-103K ( He et al., 2025 ) , by about 2% in absolute improvement, when both initialized from OpenThinker3-1.5B ( Guha et al., 2025 ) , one of the current strongest 1.5B SFT LMs, and following an identical training setup. Notably, RLVE requires no benchmark-specific data, whereas DeepMath-103K was explicitly designed for mathematical reasoning. In addition, constructing RLVE-Gym is substantially more cost-efficient, as DeepMath-103K required roughly $138,000 USD and 127,000 GPU hours to build ( He et al., 2025 ) .

We call on the community to advance research on adaptive environments, where data collection is inherently scalable and the difficulty is unbounded, with supervision signals continuously adapting right at the model’s capability frontier. We believe that environment engineering will become as foundational to LM development as feature, data, and prompt engineering, and that this work is part of a broader effort to scale RL training through adaptive environments.

## 2 Methodology

We introduce RLVE ( R einforcement L earning with Adaptive V erifiable E nvironments), an approach for scaling up LM RL training using procedurally generated data from verifiable environments, as defined in Section 2.1 . RLVE dynamically generates problems from these environments during training and can be paired with any RL algorithm that uses environment-supplied rewards. Importantly, RLVE makes each environment adaptive, adjusting its problem difficulty distribution based on the evolving capabilities of the trained policy model, as detailed in Section 2.2 .

### 2.1 Verifiable Environment

We define a verifiable environment as a tuple E = ( I , 𝒫 , R ) E=(I,\mathcal{P},R) , where I I is an input template , 𝒫 \mathcal{P} is a problem generator , and R R is a verifier (reward function) . The problem generator 𝒫 \mathcal{P} procedurally samples problems that instantiate the input template I I to produce inputs. The verifier R R is algorithmically defined, ensuring verifiable reward computation. Both 𝒫 \mathcal{P} and R R are implemented as programs. Figure 1 (b) illustrates an example of a verifiable environment that asks the LM to sort a given array in ascending order. We also define an integer difficulty level d ∈ [ 0 , + ∞ ) d\in[0,+\infty) for each environment to control the expected reasoning complexity for solving the generated problems. For example, in the sorting environment, a larger d d results in a longer array, as sorting a longer array typically requires stronger long-horizon reasoning.

Formally, a concrete problem is specified by parameters p p with environment-specific components, e.g., the length and elements of the array to be sorted in a specific problem. The parameters p p are randomly generated by the problem generator 𝒫 d \mathcal{P}_{d} conditioned on a specific difficulty level d d , written as p ∼ 𝒫 d p\sim\mathcal{P}_{d} as the program 𝒫 d \mathcal{P}_{d} defines an implicit parameter distribution. How d d influences the parameters p p sampled from 𝒫 d \mathcal{P}_{d} is specific to each environment. We denote this specific problem by E p = ( I p , R p ) E_{p}=(I_{p},R_{p}) , where I p I_{p} is the instantiated input obtained by filling the template with p p , and R p R_{p} is the corresponding verifier that computes a scalar reward R p ​ ( o ) ∈ ℝ R_{p}(o)\in\mathbb{R} for an output o o to this specific problem.

The verifier extends the range of rewards adopted in prior works on RLVR ( Lambert et al., 2025 ; DeepSeek-AI, 2025 ; Kimi Team, 2025b ) , as each environment independently defines its own verifier. For example, in an environment that asks the LM to solve a Sudoku puzzle, the verifier checks that the output satisfies all Sudoku constraints, rather than comparing it against a pre-computed correct solution, of which multiple, and sometimes many, may exist.

### 2.2 RL with Adaptive Verifiable Environments

When training a policy model π \pi , RLVE maintains a difficulty range [ ℓ π , h π ] [\ell_{\pi},h_{\pi}] that governs problem generation within a specific verifiable environment, and dynamically adjusts this range based on the current performance of π \pi . When π \pi performs well at the difficulty level h π h_{\pi} , h π h_{\pi} is incremented to shift the difficulty distribution, as shown in Figure 2 .

Specifically, we initially set ℓ π = h π = 0 \ell_{\pi}=h_{\pi}=0 , so training begins with the simplest problems available in the environment. When generating a new problem, a difficulty level d d is uniformly sampled from [ ℓ π , h π ] [\ell_{\pi},h_{\pi}] , and the environment’s problem generator samples parameters p ∼ 𝒫 d p\sim\mathcal{P}_{d} to instantiate a concrete problem. We track two quantities: the number of correct rollouts a a , and the total number of attempted rollouts b b across all problems sampled from 𝒫 h π \mathcal{P}_{h_{\pi}} . Whenever b b exceeds a minimum sample threshold τ num \tau_{\mathrm{num}} , RLVE compares the observed accuracy a / b a/b against a predefined performance threshold τ acc \tau_{\mathrm{acc}} . If a / b ≥ τ acc a/b\geq\tau_{\mathrm{acc}} , the model π \pi is considered proficient at this difficulty level, and the upper bound is incremented by one, i.e., h π ← h π + 1 h_{\pi}\leftarrow h_{\pi}+1 , thereby introducing more challenging problems. After this check, the statistics ( a , b ) (a,b) are reset, and the process continues.

RLVE does not impose a predefined cap on the upper bound h π h_{\pi} : within the available compute budget, h π h_{\pi} naturally increases as long as π \pi continues to satisfy the performance criterion at successively higher difficulty levels. To prevent unbounded expansion of the difficulty range, which would reduce exposure to harder problems, RLVE uses a sliding window of difficulty levels by capping ℓ π \ell_{\pi} with a hyperparameter d Δ > 1 d_{\Delta}>1 : after each update of the upper bound h π h_{\pi} , we set the lower bound to ℓ π = h π − d Δ + 1 \ell_{\pi}=h_{\pi}-d_{\Delta}+1 whenever the range h π − l π + 1 h_{\pi}-l_{\pi}+1 exceeds d Δ d_{\Delta} . Intuitively, RLVE exposes π \pi to problems that are neither too easy nor too hard, since it has performed well on 𝒫 h π − 1 \mathcal{P}_{h_{\pi}-1} but not yet on 𝒫 h π \mathcal{P}_{h_{\pi}} .

RLVE naturally extends training from a single verifiable environment to multiple environments jointly (Algorithm 1 ). Specifically, given a collection of n n verifiable environments { E ( 1 ) , E ( 2 ) , … , E ( n ) } \{E^{(1)},E^{(2)},\dots,E^{(n)}\} , RLVE first draws an environment E ( i ) E^{(i)} uniformly from this collection when generating a new training problem; RLVE then follows the identical algorithm described above for the selected environment E ( i ) E^{(i)} . For each adaptive environment, RLVE maintains an independent difficulty range [ ℓ π ( i ) , h π ( i ) ] [\ell_{\pi}^{(i)},h_{\pi}^{(i)}] , along with the statistics ( a ( i ) , b ( i ) ) (a^{(i)},b^{(i)}) for monitoring the model’s performance at its current upper bound difficulty level h π ( i ) h_{\pi}^{(i)} . In Section 3 , we describe how the verifiable environments are constructed.

### 2.3 RL Algorithm

Any algorithm applicable to RLVR can be directly applied to RLVE . We adopt the DAPO algorithm ( Yu et al., 2025 ) , which is a variant of the GRPO algorithm ( Shao et al., 2024 ) . We use the standard practice of DAPO’s dynamic sampling: at each rollout step, we oversample rollouts using a prompt batch size larger than the training batch size and discard prompts with identical rollout rewards across all outputs; this process repeats until the training batch is fully populated. Additional details are provided in Appendix C .

In this context, we define the effective prompt ratio as the percentage of prompts whose rollouts yield non-identical rewards (and are therefore not discarded by DAPO’s dynamic sampling). A high effective prompt ratio indicates that a large portion of problems are appropriately challenging for the policy. A lower ratio increases the time per training step, as dynamic sampling requires sending more prompts to the inference engine to obtain a single one that contributes to parameter updates; a higher ratio improves learning efficiency by reducing wasted rollouts from the inference engine, which typically constitutes the computational bottleneck in LM RL training ( Hu et al., 2024 ) .

## 3 RLVE-Gym : A Suite of 400 Environments Created through Environment Engineering

To implement RLVE , we carefully construct a suite of 400 verifiable environments, named RLVE-Gym . We give some representative sources of environments from RLVE-Gym and corresponding example environments in Table 1 . We next introduce the principles and insights used in our environment engineering , with more details in Appendix B .

### 3.1 Environments as Pedagogical Tools

The first principle we used in environment engineering is to build pedagogical tools for developing reasoning capabilities. For example, if our goal were purely to obtain a sorted array, we would execute a sorting program, which is reliable and far more efficient than relying on the model’s error-prone reasoning. Instead, we train the model to learn the reasoning process that underlies array sorting, rather than aiming to replace the sorting program with the model.