RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments — part 3/5

Cleaned-text code points [30680, 45964). Bibliography omitted here; source.md includes it.

Prior works have trained game agents ( Cobbe et al., 2020 ) and, more recently, LMs ( Hu et al., 2025c ; Chen et al., 2025a ; Liu et al., 2025d ; Stojanovski et al., 2025 ; Li et al., 2025 ) via RL on procedurally generated data. However, they employ verifiable environments with static problem difficulty distributions, which, as shown by us, either suffer from early saturation (as in Li et al. (2025) ) or inefficient learning. Among these works, Li et al. (2025) also observed that adding more environments improves performance, but their evaluations do not guarantee that test problems come from environments unseen during training; we show that such improvements generalize to entirely unseen environments. Liu et al. (2025c) trains LMs to solve SAT problems ( Cook, 1971 ) with adaptive difficulty evolving with model performance. However, we have shown that only one environment is insufficient for developing generalizable reasoning capabilities. We therefore construct RLVE-Gym to scale up the collection of training environments.

#### Adaptive data for LM training.

Curriculum learning has long been applied to improve RL efficiency by adjusting the data difficulty in increasing order throughout training ( Baker et al., 2020 ; Wang et al., 2019 ; Portelas et al., 2020 ; Jiang et al., 2020 ; Gao et al., 2025 ) , and has recently been extended to LMs ( Kimi Team, 2025b ; Shi et al., 2025 ; Chen et al., 2025b ) . These approaches operate on a finite dataset by reordering existing problems post hoc, whereas RLVE predefines difficulty levels over an infinite problem set from each verifiable environment and progresses successively through them. Cui et al. (2025) ; Yu et al. (2025) dynamically filter out prompts that do not contribute to parameter updates. These approaches operate based on the results obtained after the rollout, complementing RLVE , which adapts the problems before being sent to the inference engine. It has also been explored that a trained and increasingly stronger LM makes the environment increasingly challenging in a self-play manner ( Zhao et al., 2025 ; Liu et al., 2025a ; Liu et al., 2025b ) . In contrast, RLVE grounds environment adaptation in controllable manual construction, avoiding potentially incorrect LM-generated problems or verifiers. Besides RL training data, some work also tailors SFT data to model capability ( Khan et al., 2025 ; Zeng et al., 2025 ) .

## 7 Discussion on Future Work

#### Scaling up RL with adaptive non-verifiable environments.

While this work focuses on verifiable environments, an equally important direction is to explore adaptive non-verifiable environments, such as creative writing or deep research ( OpenAI, 2025 ; Shao et al., 2025 ) , where rewards cannot be algorithmically defined. Non-verifiable environments tend to lack clear structure, which complicates difficulty control and constitutes an open research direction. Future work should also develop systematic principles for engineering non-verifiable environments. We believe that environment engineering will become as foundational to LM development as feature, data, and prompt engineering.

#### Model-based automatic environment engineering.

In our preliminary explorations, we tried employing frontier LMs to perform automatic environment engineering. However, we found that maintaining high environment quality without human intervention proved nontrivial, particularly in ensuring (1) the unambiguity of the input template, (2) the reliability and efficiency of the problem generator in producing valid and diverse problems, and (3) the robustness of verifiers against diverse model outputs. For example, LMs often struggle to design environments that exploit the asymmetry between solving and verification in computational complexity, as such designs typically requires expert knowledge and deliberate engineering. We therefore consider the substantial human effort our authors devoted to manually engineering all 400 environments to be worthwhile. RLVE-Gym can serve as a prototype for future research on automatic environment engineering, analogous to how prior studies employed LMs for automatic data generation starting from a seed dataset ( Wang et al., 2023 ) .

## Acknowledgements

We thank Xiaoyu Chen, Zirui Cheng, Nouha Dziri, Scott Geng, Victoria Graf, Ronan Le Bras, Rulin Shao, Yijia Shao, Yizhong Wang, Teng Xiao, and Rui Xin for the helpful discussions. We thank Jiajun Li, Yuzhen Zhou, and Zilin Zhu for their help with implementing our experiments using the slime framework. We also thank Shizhe Diao and Jian Hu for providing the checkpoint obtained by continuing the original RLVR training of ProRL-1.5B-v2 on the ProRL dataset. ZZ and YW are supported by Amazon AI Ph.D. Fellowships. SSL is supported by the Meta AI Mentorship program. JH is supported by an NSF Graduate Research Fellowship and the Meta AI Mentorship program. This work is also supported by NSF Grant Nos. IIS2142739, IIS2044660, and CHE2505932; by the Defense Advanced Research Projects Agency’s (DARPA) SciFy program (Agreement No. HR00112520300); by the Singapore National Research Foundation and the National AI Group in the Singapore Ministry of Digital Development and Information under the AI Visiting Professorship Programme (award number AIVP-2024-001); by the AI2050 program at Schmidt Sciences; by a Google ML and Systems Junior Faculty Award; by gift funding from Ai2; by an Amazon AICE award; by the UW–Amazon Science Gift Hub; and by the UW–Tsukuba Amazon NVIDIA Cross Pacific AI Initiative (XPAI).

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

## Appendix A Details of RLVE

The pseudocode for RLVE is provided in Algorithm 1 . By default, we set the accuracy threshold τ acc \tau_{\mathrm{acc}} to 0.9, the minimum sample threshold τ num \tau_{\mathrm{num}} to 8 times the number of rollouts per problem, and the sliding window size d Δ d_{\Delta} to 4.

## Appendix B Details of RLVE-Gym

### B.1 Representative Sources of Verifiable Environments with Example Environments in RLVE-Gym

In this subsection, we present six representative sources of verifiable environments from RLVE-Gym , providing one example for each source. Note that the reward range is always [ − 1.0 , + 1.0 ] [-1.0,+1.0] for each of the 400 verifiable environments.

#### Verifiable environments inspired by programming competition problems.

A considerable portion of our verifiable environments is inspired by programming competition problems. We introduce the verifiable environment BubbleSwapLowerBound_PermutationCounting as an example, which is adapted from a programming problem in the Chinese National Olympiad in Informatics 2018 (CNOI 2018) 2 2 2 https://noi.cn . The input template I I is: ⬇

In this environment, the problem specification parameters p p include the length { N } and the given permutation { P } used for the lexicographic constraint. The problem generator 𝒫 d \mathcal{P}_{d} sets the permutation length as N = d + 3 N=d+3 , where d ≥ 0 d\geq 0 is the difficulty level, and uniformly samples a permutation P P of { 1 , 2 , … , N } \{1,2,\dots,N\} . The verifier R p R_{p} computes the correct answer using the algorithm that solves the original programming problem. Given a model output o o , the verifier first attempts to extract a numeric answer from o o . If the output format is invalid or the parsed answer is not a non-negative integer, the verifier assigns a reward of − 1.0 -1.0 ; otherwise, letting x x denote the correct answer and y y the model’s predicted answer, the verifier computes the reward as R p ​ ( o ) = ( min ⁡ ( x , y ) / max ⁡ ( x , y ) ) 10 , R_{p}(o)=(\min(x,y)/\max(x,y))^{10}, which smoothly penalizes deviations from the correct answer.

#### Verifiable environments of mathematical operations.

Some verifiable environments in RLVE-Gym focus on performing fundamental mathematical operations. As an illustrative example, we introduce the verifiable environment Integral , which asks the model to compute the indefinite integral of an elementary function. The input template I I is: ⬇

In this environment, the problem generator 𝒫 d \mathcal{P}_{d} (conditioned on the difficulty level d d ) randomly generates an elementary function F ⁡ ( x ) F(x) by recursively constructing an expression tree whose node count equals d + 2 d+2 . It then uses the SymPy 3 3 3 https://www.sympy.org/en/index.html Python package to compute its derivative F ′ ​ ( x ) F^{\prime}(x) , which is substituted into the input template to obtain the instantiated input I p I_{p} . The verifier R p R_{p} parses the model’s output o o into an expression and checks whether the derivative of the predicted function equals the provided F ′ ​ ( x ) F^{\prime}(x) . If the output cannot be parsed into a valid expression, the verifier assigns a reward of − 1.0 -1.0 ; if the parsed expression correctly satisfies the condition, the reward is + 1.0 +1.0 ; otherwise, it is 0.0 0.0 . Note that this environment exploits the solving–verification asymmetry: we never need to compute the integral ourselves. If we were to instead randomly sample an elementary function f ⁡ ( x ) f(x) and attempt to compute its integral ∫ f ⁡ ( x ) ​ 𝑑 x \int f(x)\,dx directly, the resulting antiderivative might not admit a closed-form expression in terms of elementary functions; consequently, even symbolic solvers such as SymPy may be unable to produce an exact expression, whereas our verification is straightforward.

#### Verifiable environments of optimization problems.

Some verifiable environments in RLVE-Gym are designed as optimization problems. We introduce the verifiable environment PolynomialMinimum . The input template I I is: ⬇

The problem generator 𝒫 d \mathcal{P}_{d} (conditioned on the difficulty level d d ) randomly generates a polynomial of degree 2 ​ ( d + 1 ) 2(d+1) , ensuring that the coefficient of the highest-order term x 2 ​ ( d + 1 ) x^{2(d+1)} is positive so that f ⁡ ( x ) f(x) admits a global minimum. The verifier R p R_{p} first attempts to extract a numeric value from the model’s output o o ; if parsing fails, it assigns a reward of − 1.0 -1.0 . Otherwise, let x 0 x_{0} denote the model’s predicted minimizer and x 0 ′ x_{0}^{\prime} the true minimizer computed analytically or via a numerical solver. The verifier then evaluates both f ⁡ ( x 0 ) f(x_{0}) and f ⁡ ( x 0 ′ ) f(x_{0}^{\prime}) and computes the reward as R p ​ ( o ) = ( ( f ⁡ ( x trivial ) − f ⁡ ( x 0 ) ) / ( f ⁡ ( x trivial ) − f ⁡ ( x 0 ′ ) ) ) 5 R_{p}(o)=((f(x_{\mathrm{trivial}})-f(x_{0}))/(f(x_{\mathrm{trivial}})-f(x_{0}^{\prime})))^{5} , where x trivial x_{\mathrm{trivial}} denotes a simple reference point (e.g., x = 0 x=0 ). This formulation smoothly rewards outputs that approach the true minimum, encouraging the model to identify increasingly accurate minima.

#### Verifiable environments of classical algorithmic problems.

Some verifiable environments in RLVE-Gym are designed based on classical algorithmic problems. As an illustrative example, we introduce the verifiable environment Sorting , which asks the model to sort a given array of numbers in ascending order. The input template I I is: ⬇

The problem generator 𝒫 d \mathcal{P}_{d} (conditioned on the difficulty level d d ) samples parameters p p such that the array length N N is roughly proportional to 3 × 1.1 d 3\times 1.1^{d} , and randomly generates the array elements. The verifier R p R_{p} first checks the output format; if the model’s output cannot be parsed into a valid list of numbers, it assigns a reward of − 1.0 -1.0 . If the output array has an incorrect length, the reward is − 0.5 -0.5 . Otherwise, let x x denote the number of positions where the predicted array elements match the array correctly sorted by a program. The verifier then computes the reward as R p ​ ( o ) = ( x / N ) 10 R_{p}(o)=(x/N)^{10} .

#### Verifiable environments of logical puzzles.

Some verifiable environments in RLVE-Gym are designed as logical puzzles. As an illustrative example, we introduce the verifiable environment Sudoku . The input template I I is: ⬇

The problem generator 𝒫 d \mathcal{P}_{d} (conditioned on the difficulty level d d ) samples parameters p p such that the larger of N N and M M does not exceed d + 2 d+2 . It exploits the solving–verification asymmetry: instead of solving Sudoku puzzles from scratch, the generator first generates a complete valid Sudoku solution by applying a sequence of random equivalence transformations (e.g., row and column swaps within bands or stacks, symbol relabeling) to a canonical solved grid. It then masks a subset of cells randomly to form the partially filled puzzle. The verifier R p R_{p} first checks the output format; if the model’s output cannot be parsed into a valid grid of the expected dimensions, a reward of − 1.0 -1.0 is assigned. Otherwise, if the filled grid satisfies all Sudoku rules, the verifier returns a reward of + 1.0 +1.0 ; if any rule is violated, the reward is 0.0 0.0 . The implementations of some other puzzle environments reference existing works ( Xie et al., 2024 ; Chen et al., 2025a ) .

#### Verifiable environments of NP-complete problems.

Some verifiable environments in RLVE-Gym are designed based on NP-complete problems. Here, we present an example, HamiltonianPathExistence , that asks the model to find a Hamiltonian path in a given directed graph, which guarantees the existence of a Hamiltonian path. The input template I I is: ⬇

The problem generator 𝒫 d \mathcal{P}_{d} (conditioned on the difficulty level d d ) samples parameters p p such that the number of vertices is N = d + 3 N=d+3 . It then exploits the solving–verification asymmetry: it first samples a random permutation of all vertices and adds directed edges between every pair of adjacent vertices in this permutation, thereby guaranteeing that the generated graph contains at least one Hamiltonian path; additional edges are then randomly added to the graph. The verifier R p R_{p} first checks whether the model’s output can be parsed into a valid list of integers; if not, it assigns a reward of − 1.0 -1.0 . If the output sequence is not a valid permutation of all vertices, the reward is − 0.5 -0.5 . Otherwise, let x x denote the number of consecutive vertex pairs in the predicted path whose corresponding directed edges actually exist in the graph; the verifier then computes the reward as R p ​ ( o ) = ( x / ( N − 1 ) ) 5 R_{p}(o)=(x/(N-1))^{5} , which equals 1.0 1.0 if and only if the path is a valid Hamiltonian path. We frequently exploit this solving–verification asymmetry when constructing verifiable environments of NP-complete problems, given that verification can be performed efficiently, while finding a valid solution remains intractable within current human knowledge.

### B.2 Full List of 400 Verifiable Environments from RLVE-Gym

We provide the complete list of all 400 verifiable environments included in RLVE-Gym in this subsection. Their detailed implementations are available in our public repository. We list the unique symbolic names of all 400 environments below.