1]University of Washington 2]Stanford University 3]Northeastern University 4]Carnegie Mellon University 5]Massachusetts Institute of Technology 6]National University of Singapore 7]Seoul National University 8]Stevens Institute of Technology 9]University of Chicago \contribution [*]Equal contribution \contribution [†]Joint last author \correspondence , , \metadata [Code] https://github.com/spade-rl/spade \metadata [Project] https://spade-rl.github.io # SPADE ♠ \spadesuit : Self-Play in Adaptive Synthetic Executable Environments ###### Abstract Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset() / step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent ’s regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent’s capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by + 5.3 +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by + 5.7 +5.7 on BFCL v4 multi-turn and + 13.9 +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement. ## 1 Introduction Agentic AI has become broadly capable: language models now reason over long horizons, use tools, search the web, and operate computers ( OpenAI, 2024 ; DeepSeek-AI, 2025 ; Wang et al., 2025c ; Zhang et al., 2025c ) . These gains increasingly come not from pretraining alone but from learning through experience, where a model improves from its own trajectories of interacting with an environment and the rewards they return ( Silver and Sutton, 2025 ) . As high-quality human text is a finite resource ( Villalobos et al., 2024 ) , the bottleneck for further agentic progress is increasingly the supply of training environments: interactive tasks with verifiable rewards. Building them has become a central industry investment, with major labs weighing environment budgets exceeding $1 billion a year and a new class of startups raising nine figures to supply them ( Zeff, 2025 ; PYMNTS, 2026 ) . Yet whether hand-built or synthesized, these environments form a fixed pool that does not adapt as the learner improves, so an agent stops improving once it exhausts them. Several lines of work try to keep agents improving, each with its own limitation. Harness engineering improves how an agent acts within a given environment at inference time, through tool orchestration and self-correction ( Yao et al., 2023 ; Shinn et al., 2023 ) , but it changes no weights, so its gains are tied to each hand-built harness, rather than improving general capabilities the model carries to new tasks. Human-curated scaling ( Zhang et al., 2025a ; Guertler et al., 2025 ; Liu et al., 2025c ) builds diverse environments to train across, but scales only as fast as people can write them. Synthetic generation ( Wang et al., 2026b ; Tu et al., 2026 ; Zhu et al., 2026 ; Gandhi et al., 2026 ; Zeng et al., 2025 ) produces environments programmatically, but the generators are fixed, so the environment space does not grow with the agent and the model soon exhausts it ( Song et al., 2024 ) . Self-play ( Zhao et al., 2025 ; Huang et al., 2025 ; Liu et al., 2025a ) lets a model improve through dual roles, but ungrounded self-play is bounded by information symmetry, unable to pose challenges beyond its own knowledge and prone to amplifying its errors ( Chae et al., 2025 ) ; corpus grounding mitigates this ( Liu et al., 2025b ) , yet most methods still generate tasks (a problem with a sparse terminal reward) rather than complete multi-turn environments (state transitions, reward functions, and verification code). None produces a self-improving system whose environments keep growing in complexity as the agent improves. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a framework where a single LLM plays two roles: an Environment Designer that produces complete training environments as executable Python code, and a Reasoning Agent that learns from them. Each environment implements a Gym-style interface (the standard reset() / step() API used in reinforcement learning) ( Brockman et al., 2016 ) , representing a full Markov decision process (MDP) with state transitions and reward functions. This code-as-environment representation unifies single-turn settings (one step to terminal reward) and multi-turn agentic tasks (sequential interaction with transition dynamics) under a single interface. Because any computable MDP can be written as a program, the Environment Designer can express any such environment in code, rather than only those a hand-designed parameterization allows. Unlike prior works where the environment generator is frozen, SPADE ’s Environment Designer is itself trained via RL with a hint-based regret signal that targets solvable environments precisely at the frontier of the Reasoning Agent ’s capability, creating co-evolution where the environment distribution shifts as the Reasoning Agent improves (Figure 2 ). Overall, our work makes the following contributions: 1. We introduce a general framework for co-evolving environments synthesis and agentic capability through self-play , where the same LLM both generates executable environments as code and learns to solve them. By representing environments as Python programs with a Gym-style interface, the framework unifies single-turn reasoning and multi-turn agentic tasks, and turns environment design into a learnable, RL-trained component of post-training, enabling continual open-ended self-improvement. 2. We introduce a hint-based regret reward for the Environment Designer . The Environment Designer is rewarded by the gap between Reasoning Agent performance with and without privileged hints, a lightweight estimate of minimax regret ( Dennis et al., 2020 ) that targets environments at the frontier of the Reasoning Agent ’s capability. Unlike a pure adversary, which is free to make environments unsolvable, or a cooperative designer, which can inflate Reasoning Agent reward without teaching it anything, hint-based regret yields constrained competitive dynamics that reward environments which are both solvable and at the learning frontier, with an equilibrium analysis in Appendix E . 3. We design a corpus-grounded and memory-augmented design pipeline spanning cognitive-skill games and tool-use tasks. Where prior environment-generation work targets a single domain (math, code, or tool use), SPADE demonstrates improvements in both settings, with environments grounded in pretraining corpus knowledge and accumulated experience. 4. We validate self-play working at 30B+ scale with practical recipe provided, demonstrating that the approach works beyond the small-model with a complete training recipe including environment validation, reward hacking avoidance, and curriculum design. On the games setting, SPADE outperforms fixed-environment baselines on all three backbones, by + 5.3 +5.3 points on average and up to + 7.5 +7.5 on individual benchmarks over the strongest fixed-environment baseline at 30B-A3B. In the tool-use setting, SPADE lifts BFCL v4 multi-turn ( Patil et al., 2025 ) by + 10.3 +10.3 at 4B and + 5.7 +5.7 at 30B-A3B, and ACEBench-Agent ( Chen et al., 2025a ) by + 13.9 +13.9 at 30B-A3B. Ablations show the advantage of hint-based regret over an exponential-moving-average (EMA) learning-potential signal ( Kanitscheider et al., 2021 ; Zhang et al., 2023 ) , and of the full adaptive, corpus-grounded, memory-augmented configuration over the partial and non-adaptive controls. Qualitative analysis reveals emergent curricula: SPADE progresses from simple single-skill tasks to complex multi-constraint environments requiring long-horizon interaction. In SPADE , the Environment Designer writes a new, self-contained environment and the Reasoning Agent learns by acting in it. Making design a learnable role lets the two co-evolve: as the agent improves, so do the environments it trains on (Figure 2 ). A single model thus improves not only how it reasons and acts, but the worlds it builds to learn in, a concrete step toward open-ended, continual self-improvement. ## 2 Related Work For an extended background and comparison to the literature see Appendix H , summarized here: Self-Play for LLMs Self-play has driven capability growth from TD-Gammon ( Tesauro et al., 1995 ) through AlphaZero ( Silver et al., 2017 ; Silver et al., 2018 ) to modern game AI ( Vinyals et al., 2019 ; † et al.(2022)(FAIR)†, Bakhtin, Brown, Dinan, Farina, Flaherty, Fried, Goff, Gray, Hu, et al., FAIR ) . PowerPlay ( Schmidhuber, 2013 ) proposed a single self-modifying system that continually invents tasks at its own capability frontier, prefiguring the dual-role single-LLM pattern adopted below. Asymmetric self-play ( Sukhbaatar et al., 2017 ) later formalized a proposer-solver paradigm in which one agent sets challenges while another solves them. Applying self-play to LLMs is harder than in single-task game-playing systems: the goal is general capability across open-ended problems, and self-play training at LLM scale is often unstable. Earlier LLM self-play methods rely on curated seed data or evaluation sets, either for self-distillation ( Chen et al., 2024 ; Yuan et al., 2024 ; Singh et al., 2023 ) , for adversarial language games over fixed vocabularies ( Cheng et al., 2024 ) , or for bootstrapping a learned proposer ( Fang et al., 2025b ; Sundaram et al., 2026 ) . A more recent line removes this dependency, training proposer and solver from minimal seeds with proxy rewards that target the learnable frontier. AZR ( Zhao et al., 2025 ) , SQLM ( Chen et al., 2025c ) , and Language Self-Play ( Kuba et al., 2025 ) have a single LLM play both roles with shared parameters; R-Zero ( Huang et al., 2025 ) and Tool-R0 ( Acikgoz et al., 2026 ) instead train two separate models initialized from the same base, using uncertainty- or solve-rate-based rewards. The closest predecessors to our work are SPIRAL ( Liu et al., 2025a ) , which trains LLMs through self-play on multi-turn zero-sum language games, and SPICE ( Liu et al., 2025b ) , which extends self-play to corpus-grounded question generation. However, both generate tasks (a single problem with only a terminal reward) and can collapse when the model’s capacity bounds the reachable solution distribution ( Chae et al., 2025 ) or when proxy rewards induce degenerate outputs ( Shafayat et al., 2025 ) ; a recent theorem-proving method adds a frozen Guide to score the proposer’s conjectures and curb collapse (SGS, Bailey et al., 2026 ); in SPADE the verifier is instead part of each generated environment, co-evolving with the Environment Designer rather than fixed. By contrast, SPADE generates full multi-turn MDP environments (state transitions, reward functions, and verification code) as executable code, trains the Environment Designer itself via RL with hint-based regret grounded in measured Reasoning Agent return, and spans a broader range of tasks (games and tool use) than prior methods confined to a single domain. Unsupervised Environment Design and Open-Endedness Curriculum learning and learnability-driven adaptation ( Bengio et al., 2009 ; Schmidhuber, 2013 ) laid the groundwork for what Dennis et al. (2020) termed unsupervised environment design (UED). POET ( Wang et al., 2019 ) pioneered paired open-ended environment-agent co-evolution via evolutionary search over parameterized terrain, while PAIRED ( Dennis et al., 2020 ) introduced adversarial environment design with minimax regret; subsequent work improved level curation via prioritized replay ( Jiang et al., 2021b ; Jiang et al., 2021a ) , evolutionary complexity ( Parker-Holder et al., 2022 ) , stability ( Mediratta et al., 2023 ) , regret approximations ( Rutherford et al., 2024 ) , and theoretical foundations ( Monette et al., 2025 ) . Güzel et al. (2025) extends prioritized level replay to imagined trajectories from a learned diffusion world model. Multi-agent autocurricula provide a complementary route to emergent complexity ( Baker et al., 2019 ) , and open-endedness has been argued essential for superhuman AI ( Hughes et al., 2024 ) . OMNI-EPIC ( Faldor et al., 2024 ) represents environments as executable code generated by foundation models, with a frozen FM judge of interestingness gating task acceptance. PAPRIKA ( Tajwar et al., 2025 ) brings curriculum-style training to LLM agents across diverse synthetic task groups. However, classical UED searches parameterized environments (maze dimensions, grid layouts) within small, fixed design spaces, and curriculum-based LLM training draws from fixed task pools without a learned generator. SPADE carries the regret principle into an unbounded, code-defined environment space: the Environment Designer is itself the learning agent, writing full MDPs as programs and co-evolving with the Reasoning Agent online, rather than an adversary selecting from a hand-designed parameterization. This turns environment design from selection within a fixed space into open-ended generation. Environment Synthesis and Scaling Following the shift from single-turn reinforcement learning with verifiable rewards (RLVR) ( OpenAI, 2024 ; DeepSeek-AI, 2025 ) to multi-turn agentic RL ( Wang et al., 2025c ; Zhang et al., 2025c ) , the env-pool bottleneck has motivated work that synthesizes training environments for LLM agents at scales unreachable by hand-curation. Recent systems generate agentic worlds and toolsets ( Wang et al., 2026b ; Tu et al., 2026 ; Dong et al., 2026 ) , terminal and computer-use sandboxes ( Zhu et al., 2026 ; Gandhi et al., 2026 ; Fan et al., 2026 ; Pi et al., 2026 ; Xue et al., 2026a ; Xue et al., 2026b ; Zhang et al., 2026e ) , and dynamic RL pipelines ( Wang et al., 2026a ; Guo et al., 2025 ; Shi et al., 2026 ) ; AgentScaler ( Fang et al., 2025a ) clusters thousands of real APIs into per-domain Python tool environments and trains agents via filtered supervised fine-tuning (SFT), and Eurekaverse ( Liang et al., 2024 ) provides a robotics analogue. Complementary task-level synthesis grounds generation in agent exploration ( Ramrakhya et al., 2025 ) , converts unverifiable text into RLVR data ( Lu et al., 2026 ) , or expands seed task spaces ( Jiang et al., 2025b ; Chen et al., 2025e ; Li et al., 2025 ; Lu et al., 2025 ) . Scaling studies show that distributions of verifiable environments lift generalization: WebScale-RL ( Cen et al., 2025 ) mines RL tasks from web pretraining data, AgentRL ( Zhang et al., 2025a ) unifies multi-turn multi-task training, and RLVE ( Zeng et al., 2025 ) hand-engineers 400 verifiable environments with adaptive difficulty levels. Curriculum selection over fixed pools provides difficulty adaptation without changing the underlying environments ( Chen et al., 2025d ; Xu et al., 2026a ) . Across these systems the environment generator is typically frozen, hand-engineered, or trained on a separate signal. SPADE instead trains the Environment Designer online, together with the Reasoning Agent , via RL with hint-based regret, produces full multi-turn MDP environments as executable Python rather than tasks with sparse terminal rewards, and co-evolves the environment distribution with the Reasoning Agent ’s capability frontier in a single training loop. ## 3 Preliminaries SPADE builds on two standard components, which we review before presenting the method: the Markov decision process with a Gym-style reset() / step() interface, used to represent environments, and reinforcement learning from verifiable rewards optimized with GRPO, used to train the policy. ### 3.1 Markov Decision Processes and Gym-Style Environments A Markov Decision Process (MDP) is a tuple ( 𝒮 , 𝒜 , T , R , ρ 0 ) (\mathcal{S},\mathcal{A},T,R,\rho_{0}) with state space 𝒮 \mathcal{S} , action space 𝒜 \mathcal{A} , transition function T ⁡ ( s ′ ∣ s , a ) T(s^{\prime}\mid s,a) , reward function R ⁡ ( s , a ) R(s,a) , and initial state distribution ρ 0 \rho_{0} . The standard programmatic interface for an MDP, popularized by OpenAI Gym ( Brockman et al., 2016 ) and standardized in its Gymnasium fork ( Towers et al., 2026 ) , exposes two functions: reset() returns an initial observation s 0 ∼ ρ 0 s_{0}\sim\rho_{0} (with an info dictionary), and step ( a a ) advances the state via T T and returns ( s ′ , r , terminated , truncated , info ) (s^{\prime},r,\text{terminated},\text{truncated},\text{info}) , separating natural termination from time-limit truncation. Our generated environments are fully observed: the observation returned at each step is the state itself. Listing 3 shows a minimal instantiation. ### 3.2 RLVR and GRPO We train the policy π θ \pi_{\theta} , an LLM with parameters θ \theta , via reinforcement learning with verifiable rewards (RLVR) using Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) . For each input prompt x x , GRPO samples a group of G G responses { y 1 , … , y G } ∼ π θ ( ⋅ ∣ x ) \{y_{1},\ldots,y_{G}\}\sim\pi_{\theta}(\cdot\mid x) and computes group-normalized advantages: A ^ i = r i − mean ​ ( { r j } j = 1 G ) std ​ ( { r j } j = 1 G ) \hat{A}^{i}=\frac{r^{i}-\text{mean}(\{r^{j}\}_{j=1}^{G})}{\text{std}(\{r^{j}\}_{j=1}^{G})} (1) The policy is updated via clipped policy gradient with KL regularization: ℒ ( θ ) = − 1 G ∑ i = 1 G min ( π θ ​ ( y i ∣ x ) π old ​ ( y i ∣ x ) A ^ i , clip ( π θ ​ ( y i ∣ x ) π old ​ ( y i ∣ x ) , 1 − ε low , 1 + ε high ) A ^ i ) + β KL ⋅ KL [ π θ ∥ π ref ] \mathcal{L}(\theta)=-\frac{1}{G}\sum_{i=1}^{G}\min\left(\frac{\pi_{\theta}(y_{i}\mid x)}{\pi_{\text{old}}(y_{i}\mid x)}\hat{A}^{i},\;\text{clip}\!\left(\frac{\pi_{\theta}(y_{i}\mid x)}{\pi_{\text{old}}(y_{i}\mid x)},1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}}\right)\hat{A}^{i}\right)+\beta_{\text{KL}}\cdot\text{KL}[\pi_{\theta}\|\pi_{\text{ref}}] (2) ## 4 SPADE : Self-Play in Adaptive Synthetic Executable Environments SPADE is an end-to-end self-play framework where a single LLM, π θ \pi_{\theta} , alternates between generating executable training environments and learning to solve them (Figure 4 ; Algorithm 1 ). Its three components create an adaptive curriculum: § 4.1 details how dual-role self-play couples environment generation with Reasoning Agent (RA) learning; § 4.2 explains how hint-based regret steers the Environment Designer (ED) toward environments at the Reasoning Agent ’s learning frontier; and § 4.3 outlines the pipeline that grounds and validates these executable environments. ### 4.1 Dual-Role Self-Play and Code-as-Environment SPADE formulates adaptive environment self-play as a game where a single model π θ \pi_{\theta} acts in two roles via role-specific system prompts: designing environments ( role = D \text{role}{=}D ) or solving them ( role = A \text{role}{=}A ). Let ℰ \mathcal{E} denote the space of all valid environments. In the Environment Designer role ( π D \pi_{D} ), π θ \pi_{\theta} produces an executable environment e ∈ ℰ e\in\mathcal{E} as a Python program implementing the Gym-style reset() / step() API (Section 3.1 ; a minimal example is shown in Appendix J.3.1 ), where the transition function T ⁡ ( s ′ ∣ s , a ) T(s^{\prime}\mid s,a) and reward function R ⁡ ( s , a ) R(s,a) are both encoded in the step() implementation. The Environment Designer also emits a privileged hint for each environment. A hint h h is task-relevant information that the Environment Designer attaches to an environment (for example, a partial solution sketch or a key structural observation); revealing h h to the Reasoning Agent makes the environment easier to solve, and the gap in Reasoning Agent return with versus without h h defines the Environment Designer ’s hint-based regret reward (Section 4.2 ). In the Reasoning Agent role ( π A \pi_{A} ), π θ \pi_{\theta} interacts with e e via sequential actions, receiving observations and rewards. This code-as-environment representation unifies single-turn settings ( reset() → \rightarrow step(answer) → \rightarrow terminal reward) and multi-turn agentic settings ( reset() → \rightarrow step() → ⋯ → \rightarrow\cdots\rightarrow done ) under a single interface and training pipeline. This representation provides the vast task space that AI-GAs ( Clune, 2019 ) argue is necessary for open-ended self-improvement: any computable MDP can be expressed as a Python program, so the Environment Designer can express any such environment in code rather than being confined to a hand-designed environment parameterization. Both roles share parameters θ \theta , so updates for either role affect the same policy. Each self-play cycle first collects Environment Designer trajectories, and then Reasoning Agent trajectories from those generated environments. The Environment Designer receives hint-based regret (Section 4.2 ), whereas the Reasoning Agent receives task correctness from the environment’s reward function. Each candidate environment is validated for syntactic correctness and executability before entering the training pool; validation details are in Appendix F.2 . Stabilizing joint two-role training. Optimizing a single policy for two coupled objectives requires several stabilizing techniques ( Liu et al., 2026a ) . Following SPIRAL ( Liu et al., 2025a ) , we normalize advantages independently for each role. We standardize Reasoning Agent returns within each environment’s rollouts and mean-center Environment Designer rewards within each skill. To ensure both roles contribute comparably, we upweight the less frequent Environment Designer trajectories. Finally, we delay the Environment Designer update by k k rollouts, where k k is the number of training steps per environment set, so the difficulty anchor can score each environment’s Reasoning Agent win rate over the full training window; the regret component is computed at generation time. Since this delay makes the Environment Designer objective off-policy, we correct the gradient using truncated importance sampling ( Yao et al., 2025 ) . We use an asymmetric clipping range ( ε low = 0.2 \varepsilon_{\text{low}}{=}0.2 , ε high = 0.28 \varepsilon_{\text{high}}{=}0.28 ) to preserve exploration and floor the raw regret at zero. The deployed Environment Designer reward blends this floored regret, normalized to [ 0 , 1 ] [0,1] by a fixed scale, (weight 0.4 0.4 ) with a flat-top difficulty anchor that pays environments whose Reasoning Agent win rate falls in a target band and decays linearly outside it (weight 0.6 0.6 , band [ 0.4 , 0.6 ] [0.4,0.6] ): the anchor regulates difficulty, and floored regret selects the most teachable environments within the band. Full hyperparameters appear in Appendix F.2 . ### 4.2 Hint-Based Regret Reward The Environment Designer ’s reward for producing environment e e is: r D ​ ( e ) = r ¯ A ​ ( e ∣ h ) − r ¯ A ​ ( e ) , r_{D}(e)=\bar{r}_{A}(e\mid h)-\bar{r}_{A}(e), (3) where r ¯ A ​ ( e ∣ h ) = 1 G ​ ∑ i = 1 G r A ​ ( y i ′ ∣ e , h ) \bar{r}_{A}(e\mid h)=\frac{1}{G}\sum_{i=1}^{G}r_{A}(y^{\prime}_{i}\mid e,h) is the Reasoning Agent ’s average return over G G fresh rollouts y i ′ y^{\prime}_{i} sampled with the privileged hint h h in context, and r ¯ A ​ ( e ) \bar{r}_{A}(e) is the corresponding average over G G rollouts without it. Three regimes emerge: high regret indicates an environment at the learning frontier (solvable with hints but not without); low regret where the Reasoning Agent succeeds with and without the hint indicates mastery; and low regret with low returns even with the hint indicates an intractable environment. This implements the minimax regret objective of PAIRED ( Dennis et al., 2020 ) without a separate antagonist: the hint-equipped Reasoning Agent serves as the upper-bound policy. For a policy that best-responds to the hint, extra information cannot reduce expected return, so regret is non-negative at the optimum; for the current policy, expected regret can dip below zero when hints mislead it, which we observe for the smaller backbones (Figure 13 ). We formalize this intuition and show that, under idealized assumptions, every pure Nash equilibrium yields hint-free optimality on every valid environment (Appendix E ). We compare hint-based regret against an EMA-based learning-potential signal ( Kanitscheider et al., 2021 ; Zhang et al., 2023 ) in the Environment Designer -reward ablation (Section 7.2 ). Hint generation. For each environment e e , the Environment Designer (same LLM, different prompt) emits one task-specific hint of a few sentences. The prompt asks for the key insight or strategy and the expected answer format, but forbids revealing the exact answer; its output is unlabeled hint text. Figure 5 shows two high-regret examples. Each hint is shown beside the task it conditions and one logged Reasoning Agent rollout from each prompt condition. The paired trajectories expose the behavioral source of the return gap: the hint hands the fiber-task agent the probe pattern almost outright, and narrows the audio-task agent’s search to a parameter range it would otherwise reach only after repeated failed attempts. ### 4.3 Environment Design Corpus grounding. A generator conditioned only on its own output has no source of novelty outside its own weights, so it narrows toward the patterns it already favors and mode-collapses, the “invisible leash” ( Chae et al., 2025 ; Zhang et al., 2026c ; Jiang et al., 2025a ) . We treat an external corpus as the mechanism that lengthens that leash: every round the Environment Designer conditions on freshly sampled human corpus. In the games setting these are 10 10 k mathematics and 5 5 k science documents drawn from DCLM ( Li et al., 2024 ) and MegaScience ( Fan et al., 2025 ) , spanning web pages, physics forums, and university-level scientific textbooks; in the tool-use setting, 15 15 k documents from the Nemotron pretraining code corpus ( NVIDIA, 2026 ) , which supplies algorithm implementations and API documentation. SPICE ( Liu et al., 2025b ) established corpus grounding for task generation, mining documents for reasoning questions; SPADE carries the principle to environment generation, where a sampled passage seeds an executable MDP rather than a question-answer pair. Section 6.2 (Figure 8 ) shows how corpus grounding sustains environment diversity. Environment memory. The corpus supplies breadth from outside the loop; the cross-episode memory keeps the Environment Designer from re-posing what the Reasoning Agent has already mastered ( Schmidhuber, 2013 ) . A buffer of previously generated environments, annotated with regret scores and skill tags, gives the Environment Designer high-regret seeds to vary, so each round starts from what the Reasoning Agent currently finds hard rather than from scratch. Retaining and reusing past experience this way is the mechanism agentic-memory systems rely on to keep improving without further gradient updates ( Zhang et al., 2026d ; Xiong et al., 2026 ) ; here it acts on the design side, holding generated difficulty at the Reasoning Agent ’s frontier as that frontier moves. The two inputs work on different axes, the corpus on what environments are about and the memory on how hard they are, and Table 3 separates their contributions. Environment pool management, lifecycle, and quality control details are in Appendix F.2 . ## 5 Experimental Setup ### 5.1 SPADE Training Recipe We train three Qwen3 backbones: Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507, the last of which is our primary model. The 4B and 30B runs use instruct models; the 8B run enables thinking for both the Environment Designer and the Reasoning Agent . We train with GRPO ( Shao et al., 2024 ) for 400 400 rollouts of 24 24 environments each. The Environment Designer regenerates the environment set every k k rollouts and its update is delayed by the same k k (Section 4.1 ), so k k sets how long the Reasoning Agent trains on a fixed set before the curriculum moves. The Reasoning Agent plays every environment 16 × k 16\times k times without the hint and 16 16 times with it; the regret subtracts hint and no-hint averages measured at the same regeneration step ( 16 16 plays each). The Environment Designer writes the privileged hint itself, with no external model, and each round it also draws high-regret seeds from a memory of past environments. Rewards are normalized per environment, and the stabilizers described earlier (per-role advantages, the delayed Environment Designer update with truncated importance sampling, and the regret floor) are unchanged across settings. Our RL backbone is slime ( Zhu et al., 2025 ) . The settings below differ in what the Environment Designer generates, how it is grounded, how the Reasoning Agent acts, and the regeneration interval k k . Every benchmark is held out from training, and component-level controls such as removing Environment Designer training or grounding are covered by the ablations of Section 7 . Full hyperparameters, per-run adjustments, and evaluation protocol details are in Appendices F.2 and F.3 . ### 5.2 Domain: Games Environments. The Environment Designer writes each environment as a self-contained Python game with a verifiable reward, and the Reasoning Agent solves it. Each rollout uses 24 24 games, eight for each of three active skills; the six cognitive-skill categories (Mathematical Reasoning, Logical Deduction, Spatial Reasoning, Pattern Recognition, Optimization, and Causal Inference) rotate three at a time, with k = 4 k{=}4 . The generation is grounded in the 15 15 k-document math and science corpus of Section 4.3 . Besides floored hint-based regret, the Environment Designer reward includes the flat-top difficulty anchor of Section 4.1 that pays games whose Reasoning Agent win rate falls in the target band, and every game must pass syntax and execution checks before it enters the pool. Baselines and evaluation. We retrain two fixed-environment baselines separately for each backbone from its corresponding base checkpoint, using the same 400 400 -iteration budget: Fixed-env RLVE , GRPO on the official RLVE set ( Zeng et al., 2025 ) , and Fixed-env GRPO , which is generated offline by that same Qwen3 backbone and held fixed throughout training. Fixed-env RLVE follows the official RLVE sampling and curriculum settings. RLVE has the higher suite average at every backbone scale. Evaluation spans two distances from the training distribution: procedural reasoning , the four Reasoning-Gym ( Stojanovski et al., 2026 ) categories at hard difficulty, and out-of-distribution (OOD) reasoning and code , AIME 2025/2026 (Avg@32) ( MAA, ) , GPQA-Diamond (accuracy) ( Rein et al., 2023 ) , and LiveCodeBench-v6 (Pass@1) ( Jain et al., 2024 ) . Table 1 reports all eight for every backbone. ### 5.3 Domain: Tool Use Environments. The Environment Designer instead writes an executable tool-use environment: a set of simulated tools in OpenAI function-calling format, a backend state the tools modify, and three to five natural-language user instructions that arrive one at a time, each with a check on the resulting state. Generation is grounded in the 15 15 k-document code corpus, and tool environments cost more to generate, so we used k = 8 k{=}8 . The Reasoning Agent solves an environment by calling tools over multiple turns and is rewarded only when it completes every user instruction. The privileged hint is a step-by-step plan (look up a record, update it, then reconcile the result), so the Reasoning Agent still has to find the exact arguments by calling the tools. Validation adds two checks beyond the game ones: a deterministic reset gate that exercises every success criterion under several seeds, and an LLM check that every success criterion can be met by some tool (details in Appendix F.2 ). The generation prompt targets the multi-turn function-calling task family of the evaluation suites (schema-defined simulated tools, a backend state, and per-instruction checks); no benchmark tasks or data are shown to the Environment Designer . Baselines and evaluation. We compare SPADE with four synthetic-environment systems: AgentScaler ( Fang et al., 2025a ) , Agent-World ( Dong et al., 2026 ) , Agent World Model (AWM) ( Wang et al., 2026b ) , and EnvScaler ( Song et al., 2026 ) . Their results are transcribed from the cited papers and differ from ours in training data and budget, in some cases base model, and in evaluation protocol (Appendix F.3.1 ). Evaluation uses BFCL v4 multi-turn ( Patil et al., 2025 ) , τ 2 \tau^{2} -bench ( Barres et al., 2025 ) , and ACEBench-Agent (the Agent category of ACEBench-en) ( Chen et al., 2025a ) , reported per domain in Table 2 . ## 6 Experimental Results We evaluate SPADE on two settings, game and tool-use environment design, across the three Qwen3 backbones of Section 5.1 . Section 6.1 reports held-out benchmark performance against the fixed-environment baselines, and Section 6.2 examines what the Environment Designer and Reasoning Agent produce over training. ### 6.1 Quantitative Analysis Synthetic game environments improve science, code, and procedural reasoning. At 30B-A3B, SPADE reaches a suite average of 58.3 58.3 : + 8.1 +8.1 over base and + 5.3 +5.3 over the strongest fixed-environment baseline (Fixed-env RLVE), a margin that grows with model size (Table 1 ). The Environment Designer trains on synthetic games, never on any held-out task, yet the gains transfer to science, code, and procedural reasoning and hold late in the 400 400 -step run (Figure 6 ), while competition math is preserved. They concentrate on procedural reasoning, where every cognitive-skill category improves: the generated games exercise each skill through many problem structures rather than a fixed task set. Synthetic tool-use environments improve multi-step interaction. The same recipe applied to tool-use environment design improves every backbone (Table 2 ), and at 30B-A3B the size of the gain tracks how closely a benchmark’s task structure matches the generated environments: ACEBench-Agent gains most ( + 13.9 +13.9 ), and its stateful, multi-step tasks mirror the generated pattern of a database, a tool schema, and a multi-call goal. BFCL v4 multi-turn follows ( + 5.7 +5.7 , and + 10.3 +10.3 at 4B), then τ 2 \tau^{2} -bench ( + 3.6 +3.6 ). SPADE at 30B-A3B leads the dedicated data-synthesis systems on both BFCL v4 multi-turn and ACEBench-Agent, so structural training signal transfers where domain-specific data collection does not reach. ### 6.2 Qualitative Analysis The Environment Designer keeps supplying environments the Reasoning Agent can learn from. We examine what each role produces over the main 30B games run. Figure 8 tracks the learnable share of each rollout, the fraction of generated environments on which the Reasoning Agent wins between 20 % 20\% and 80 % 80\% of the time. Full SPADE raises this share over training, reaching roughly a third late in the 400-step run; the corpus, the memory, and Environment Designer training together sustain that supply. Figure 8 traces the diversity of that supply to the corpus: corpus grounding keeps the environments diverse (Vendi/ n n 0.68 0.68 , versus 0.04 0.04 without it), and the control without Environment Designer training or memory, but with the corpus, retains full diversity ( 0.70 0.70 , Appendix I.1.2 ). Corpus grounding provides the breadth, while the full Environment Designer training configuration sharpens the difficulty. Ivison (2026) reaches the same conclusion from the failure side: an ungrounded proposer collapses onto a handful of near-identical programs, explicit diversity rewards get hacked once they are optimized against, and conditioning on an external corpus is the intervention that holds longest. Figure 8 shows that corpus grounding sustains diversity, whereas the ungrounded run collapses. Environment Designer training makes environments harder in measurable, code-level ways. The generation instructions are fixed across all 400 steps (the same template asking for hidden state, multi-turn structure, and partial reward), while the grounding document is resampled from the corpus each step with no trend over training, so any systematic change in the generated environments traces to Environment Designer training rather than to a prompt schedule. Two measurements move together. First, within Physics, the share of environments whose opening observation prints the governing formula falls from 25 % 25\% to 5 % 5\% over the 473 473 Physics environments of the canonical run (Figure 9 ). Second, rewards become more finely graded, from 3.7 3.7 to 5.8 5.8 distinct levels per environment, of which 2.2 → 4.0 2.2\to 4.0 are strictly partial (Appendix I.1.1 ). The no-corpus run (w/o corpus grounding in Table 3 ) shows none of this: over steps 290–312 it emits the same rotating-maze task 41 times in a row. The Reasoning Agent learns to act on evidence rather than derive in advance. Figure 10 shows the Reasoning Agent ’s side of the same run. At step 0 it reasons entirely up front and cannot recover once the interface rejects its answer; by step 200 it tests short hypotheses and revises them as results return; by step 300 it gathers evidence first and derives once, when the evidence is enough. The shift appears only where the task rewards inference: on procedural environments the same policy acts in short commands (late-step median 8 8 tokens). The model still derives at length when the task calls for it, and the late-checkpoint benchmark gains (Table 1 ) confirm it keeps the ability. ## 7 Ablations We ablate two design choices for the Environment Designer : whether it adapts during training (Section 7.1 ), and how it is rewarded (Section 7.2 ). Table 3 covers the ablation for training settings and environment sources. The first two settings remove corpus grounding and the environment memory, one at a time. The remaining two variants remove online Environment Designer training in different ways: one continues self-generation but removes both Environment Designer training and the environment memory; the other trains the Reasoning Agent on a static pool generated offline by GPT-5.5, with corpus grounding. All variants use games on Qwen3-30B-A3B-Instruct-2507 and are evaluated on the same eight-benchmark suite as the main experiments. Each variant reports its best checkpoint on the suite average; full eval curves are in Figure 11 , and per-ablation breakdowns in Appendix G . ### 7.1 Ablation: Environment Designer Adaptation As shown in Table 3 and Figure 11 , the full SPADE run is strongest across settings. Removing both Environment Designer training and memory makes the model worse than no training at all, with the eight-benchmark average falling to 40.5 40.5 , 9.7 9.7 points below the untrained base. The static GPT-5.5 pool, generated offline with corpus grounding but no environment memory, does better, beating the base ( 53.0 53.0 versus 50.2 50.2 ), but recovers only about 35 % 35\% of SPADE ’s + 8.1 +8.1 gain and does not improve code (LiveCodeBench-v6 42.6 42.6 versus 43.2 43.2 ). The partial and non-adaptive variants peak early and then fade (the no-memory and no-corpus runs near step 111, the GPT-5.5 pool near step 175), while full SPADE stays strongest late in training (Figure 11 ). Each control changes more than one factor: the self-generation control removes both Environment Designer training and memory, while the GPT-5.5 control changes the generator and uses an offline, memory-free pool. Together they show that the full co-adaptive setup in SPADE wins. ### 7.2 Ablation: Environment Designer Reward SPADE ’s Environment Designer reward is built on hint-based regret (Eq. 3 ), blended with the difficulty anchor of Section 4.1 . The alternative below is cheaper: it reuses the Reasoning Agent rollouts already collected and needs no hinted replay. EMA-based learning potential. This reward design ( Kanitscheider et al., 2021 ; Zhang et al., 2023 ) scores an environment by how far the Reasoning Agent ’s success on it departs from its skill’s recent average. Environments are grouped by skill s s . Let r ¯ A ​ ( e ) \bar{r}_{A}(e) be the Reasoning Agent ’s average return on environment e e (Eq. 3 ), and r ¯ A , t ​ ( s ) \bar{r}_{A,t}(s) its mean over the skill’s environments in scoring round t t . Two moving averages of r ¯ A , t ​ ( s ) \bar{r}_{A,t}(s) are tracked at rates γ 1 > γ 2 \gamma_{1}>\gamma_{2} : μ t γ ​ ( s ) = ( 1 − γ ) ​ μ t − 1 γ ​ ( s ) + γ ​ r ¯ A , t ​ ( s ) , μ fast ≡ μ γ 1 , μ slow ≡ μ γ 2 . \mu^{\gamma}_{t}(s)\;=\;(1-\gamma)\,\mu^{\gamma}_{t-1}(s)\;+\;\gamma\,\bar{r}_{A,t}(s),\qquad\mu_{\text{fast}}\equiv\mu^{\gamma_{1}},\quad\mu_{\text{slow}}\equiv\mu^{\gamma_{2}}. (4) Here γ \gamma weights the new observation, so μ fast \mu_{\text{fast}} adapts faster than μ slow \mu_{\text{slow}} , and μ γ \mu^{\gamma} has half-life log ⁡ ( 0.5 ) / log ⁡ ( 1 − γ ) \log(0.5)/\log(1-\gamma) rounds. The reward uses μ slow \mu_{\text{slow}} as the skill baseline: ρ ⁡ ( e ) \displaystyle\rho(e)\; = | r ¯ A ​ ( e ) − μ slow , t ​ ( s ) | , \displaystyle=\;\bigl|\,\bar{r}_{A}(e)-\mu_{\text{slow},t}(s)\,\bigr|, (5) r D LP ​ ( e ) \displaystyle r_{D}^{\text{LP}}(e)\; = ρ ⁡ ( e ) − 1 | ℰ s | ​ ∑ e ′ ∈ ℰ s ρ ⁡ ( e ′ ) , \displaystyle=\;\rho(e)-\tfrac{1}{|\mathcal{E}_{s}|}\textstyle\sum_{e^{\prime}\in\mathcal{E}_{s}}\rho(e^{\prime}), (6) where ℰ s \mathcal{E}_{s} is the round’s environment set for skill s s . Two consequences follow. The deviation is unsigned, so an environment the Reasoning Agent always solves can score as highly as one it never solves when the slow mean sits mid-range, though neither gives gradient; only a variance-style bonus such as r ¯ A ​ ( e ) ​ ( 1 − r ¯ A ​ ( e ) ) \bar{r}_{A}(e)\bigl(1-\bar{r}_{A}(e)\bigr) would favour mixed outcomes. And μ fast \mu_{\text{fast}} is tracked but unused: the classical gap | μ fast − μ slow | |\mu_{\text{fast}}-\mu_{\text{slow}}| is logged only as a diagnostic. The signal also needs a per-skill history before it is meaningful, and cannot separate deviation caused by environment design from Reasoning Agent drift or sampling noise. Results. Figure 12 shows the trajectories for the two trained- Environment Designer rewards and the control without Environment Designer training or memory on the six-benchmark trajectory average, while Table 6 gives their selected checkpoints on the full eight-benchmark suite. Hint-based regret lifts the eight-benchmark average from 50.2 50.2 to 58.3 58.3 ( + 8.1 +8.1 ); EMA-based learning potential reaches around 70 % 70\% of that gain ( + 5.7 +5.7 , to 55.9 55.9 ) and climbs more slowly, the two signals separating after the first ∼ 50 {\sim}50 steps. Both stay far clear of the control without Environment Designer training or memory, whose selected checkpoint is 9.7 9.7 points below the untrained model. The comparison between the two trained- Environment Designer variants isolates the reward choice, the regret-based blend versus the standalone learning-potential signal; the third comparison does not isolate Environment Designer training from memory. The form of Eq. 6 helps explain the remaining reward gap: an unsigned deviation from a per-skill running mean needs history before it means anything, and even then scores mastered and hopeless environments alike, so it finds the frontier later and less sharply than a hint gap measured on the current policy. Per-benchmark numbers are in Table 6 . ## 8 Scaling Results Scaling model size. Under a shared training recipe (Section 5.1 ), the average gain over each backbone’s base grows from + 5.2 +5.2 at 4B and + 5.7 +5.7 at 8B to + 8.1 +8.1 at 30B-A3B, while Fixed-env GRPO stays near + 1.2 +1.2 at every size (Figure 13 ; Table 1 ). Fixed-env GRPO supplies a static training signal, whereas the Environment Designer keeps adapting the curriculum to the Reasoning Agent ; the widening gap is consistent with larger models benefiting more from that adaptivity. Scaling curriculum diversity. The cognitive-skill curriculum is the set of skill categories the Environment Designer targets when it generates environments: the six listed in Section 5.2 , three active per regeneration in round-robin. We compare the full six-skill curriculum against a two-skill version, with everything else held fixed (Figure 14 ; per-benchmark panels in Figure 15 , Appendix G ). The two-skill run also improves, but it captures only about half of the Reasoning-Gym gains and much smaller GPQA-Diamond and LiveCodeBench-v6 gains (best checkpoint on the eight-benchmark suite of Table 1 : 53.7 vs. 58.3). The gains grow with the diversity of the curriculum, not with any single game family. ## 9 Discussion SPADE ’s results support two conclusions. First, training the Environment Designer to produce environments adapted to the Reasoning Agent ’s current ability outperforms training on fixed environment sources: both the main Fixed-env GRPO pools generated once offline by the corresponding Qwen3 backbones and held fixed throughout training (Table 1 , Section 6.1 ), and the offline pool generated by the stronger GPT-5.5 model in the ablation (Table 3 , Section 7 ). The widening gap over Fixed-env GRPO with model scale is consistent with an adaptivity advantage. Second, one code-as-environment interface spans two very different settings. In the games setting, reasoning skills the Reasoning Agent develops (planning, constraint satisfaction, strategic thinking) generalize to held-out mathematics, science, and code, well beyond the game format they were learned in. In the tool-use setting, the same recipe lifts multi-step agentic benchmarks most where interaction matters, by + 13.9 +13.9 on ACEBench-Agent and + 5.7 +5.7 on BFCL v4 multi-turn at 30B-A3B. The same learnable environment-design role is effective in both settings. Limitations. (a) Complexity bounded by scale and the invisible leash ( Chae et al., 2025 ) . The Environment Designer cannot produce environments more complex than its base model can express in context, so reachable environment complexity grows with model scale and generation budget. (b) A human-designed optimizer. Both roles are updated by a fixed, human-authored RL algorithm (GRPO); SPADE does not modify its own learning rule. (c) No formal optimality, and fixed-task evaluation. Hint-based regret is motivated by PAIRED ( Dennis et al., 2020 ) but not proven to yield an optimal curriculum, and current benchmarks measure fixed-task performance rather than open-ended reasoning growth. Future directions. We improve the Environment Designer with gradient updates, but co-adaptation between the Environment Designer and Reasoning Agent might also come from an Environment Designer that improves without weight updates, accumulating and refining design strategies from past attempts in context; whether learned weights or in-context evolution makes the better designer, and at what scale, is an open question. SPADE also automates one stage of post-training; combining adaptive environment design with systems that automate the remaining stages, from data curation through the learning rule itself, could extend co-adaptive self-play across the whole training pipeline. ## 10 Conclusion We presented SPADE , a framework that makes environment design a learnable component of LLM post-training through self-play. Three contributions enable this: (1) a hint-based regret reward that trains the Environment Designer to produce environments at the Reasoning Agent ’s learning frontier, grounded in minimax regret theory; (2) environment design anchored in a pretraining corpus and an environment memory, with a code-as-environment interface that unifies single-turn and multi-turn settings; and (3) a practical recipe at 30B+ scale on Qwen3 models. In the games setting, SPADE raises held-out reasoning and code benchmarks by + 8.1 +8.1 average points over base at 30B-A3B ( + 5.2 +5.2 and + 5.7 +5.7 at 4B and 8B); these gains remain at late checkpoints of the 400 400 -step run, and the margin over the strongest fixed-environment baseline grows with scale. The same recipe lifts multi-step tool-use benchmarks by + 13.9 +13.9 on ACEBench-Agent and + 5.7 +5.7 on BFCL v4 multi-turn. SPADE shows that a single model can design its own training environments and improve from them, a step from fixed benchmarks toward open-ended, continual self-improvement. ## Acknowledgments This research was supported by the UW-Amazon Science Gift Hub, UW-Tsukuba Amazon NVIDIA Cross Pacific AI Initiative (XPAI), Sony Research Award, Modal Research Grants, Tinker Research Grants, Character.AI, DoorDash, Open Philanthropy, Coefficient Giving, Toyota Research Institute, and the Schmidt AI2050 Fellows program. This material is based upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number(s): FA8650-23-C-7316. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL or DARPA. ## Appendix E Theoretical Analysis [back to contents] This section formally analyzes the incentive structure of the hint-based regret reward (hereafter hint-regret) introduced in Section 4.2 . We model the interaction between the Environment Designer and Reasoning Agent as a two-player game: the Environment Designer selects an environment distribution, while the Reasoning Agent selects a policy. Since LLM inference is stochastic, all payoffs are defined through expected verifier returns. The main result is an equilibrium characterization of the idealized game. If positive hint-regret remains anywhere, then the current unhinted Reasoning Agent is suboptimal there, and the Environment Designer can profitably concentrate on it. Consequently, at any pure Nash equilibrium, the Environment Designer ’s expected regret is zero, the Reasoning Agent is hint-free optimal on every environment in ℳ \mathcal{M} , and privileged hints become vacuous. ### E.1 Setup and Assumptions ##### Environment class. Let 𝒰 \mathcal{U} be a finite universe of candidate environments. The set of mathematically valid executable environments is denoted by ℳ ⊆ 𝒰 \mathcal{M}\subseteq\mathcal{U} . These are environments whose specifications are internally consistent and whose attempted solutions can be evaluated by an external verifier. In the implementation, ℳ \mathcal{M} is approximated by syntax checks, executability checks, and, in the tool-use setting, solvability filtering. ##### Distribution notation. For any finite set 𝒳 \mathcal{X} , let Δ ⁡ ( 𝒳 ) \Delta(\mathcal{X}) denote the set of probability distributions over 𝒳 \mathcal{X} . If D ∈ Δ ⁡ ( ℳ ) D\in\Delta(\mathcal{M}) (the Environment Designer ’s environment distribution; we reuse the letter D D for this distribution, while the subscript in u D u_{D} below labels the Environment Designer role) and ℬ ⊆ ℳ \mathcal{B}\subseteq\mathcal{M} , write D ⁡ ( ℬ ) := ∑ e ∈ ℬ D ⁡ ( e ) . D(\mathcal{B}):=\sum_{e\in\mathcal{B}}D(e). A property holds for D D -almost every environment if the set on which it fails has D D -mass zero. ##### Inference protocol. Fix an inference budget B inf B_{\text{inf}} . This budget includes all evaluation-time choices that affect the set of possible Reasoning Agent trajectories: model architecture, context length, prompt format, maximum generation length, sampling rule, temperature, number of rollouts, tool access, memory access, and verifier. Let 𝒴 B inf \mathcal{Y}_{B_{\text{inf}}} denote the finite trajectory space induced by this budget. ##### Fine-tuning budget. Fix a fine-tuning budget F F . This budget includes all training-time choices that determine which policies the procedure can produce: optimizer, objective, number of updates, and regularization. Let Π \Pi denote the set of policies attainable under this budget; we take Π \Pi to be finite, so maxima over Π \Pi are attained. ##### Returns with and without hints. For each environment e ∈ ℳ e\in\mathcal{M} , let h ⁡ ( e ) h(e) denote the privileged hint emitted by the Environment Designer ; hints are modeled as a fixed map e ↦ h ⁡ ( e ) e\mapsto h(e) , so strategic hint choice by the Environment Designer is outside this idealization. A Reasoning Agent policy π ∈ Π \pi\in\Pi induces a distribution over trajectories in 𝒴 B inf \mathcal{Y}_{B_{\text{inf}}} given either the unhinted input e e or the hinted input ( e , h ⁡ ( e ) ) (e,h(e)) . Let r ⁡ ( e , y ) ∈ [ 0 , 1 ] r(e,y)\in[0,1] denote the verifier return of trajectory y ∈ 𝒴 B inf y\in\mathcal{Y}_{B_{\text{inf}}} on environment e e . Returns are normalized to [ 0 , 1 ] [0,1] for the analysis; training uses a shaped reward in [ − 1 , 1 ] [-1,1] (Algorithm 1 ). Here r ⁡ ( e , y ) r(e,y) is the per-trajectory return written r A ​ ( y ∣ e ) r_{A}(y\mid e) in Section 4 , and R π ​ ( e ) R_{\pi}(e) , R π h ​ ( e ) R_{\pi}^{h}(e) below are the population analogues of the empirical means r ¯ A ​ ( e ) \bar{r}_{A}(e) and r ¯ A ​ ( e ∣ h ) \bar{r}_{A}(e\mid h) . The unhinted expected return of policy π \pi on environment e e is R π ( e ) := 𝔼 y ∼ π ( ⋅ ∣ e ) [ r ( e , y ) ] , R_{\pi}(e):=\mathbb{E}_{y\sim\pi(\cdot\mid e)}[r(e,y)], and the hinted expected return is R π h ( e ) := 𝔼 y ∼ π ( ⋅ ∣ e , h ( e ) ) [ r ( e , y ) ] . R_{\pi}^{h}(e):=\mathbb{E}_{y\sim\pi(\cdot\mid e,h(e))}[r(e,y)]. ##### Game payoffs. The Environment Designer chooses a distribution D ∈ Δ ⁡ ( ℳ ) D\in\Delta(\mathcal{M}) . Its payoff is the expected hint-based regret u D ​ ( D , π ) := 𝔼 e ∼ D ​ [ R π h ​ ( e ) − R π ​ ( e ) ] . u_{D}(D,\pi):=\mathbb{E}_{e\sim D}\left[R_{\pi}^{h}(e)-R_{\pi}(e)\right]. The Reasoning Agent ’s payoff is its unhinted expected return u A ​ ( D , π ) := 𝔼 e ∼ D ​ [ R π ​ ( e ) ] . u_{A}(D,\pi):=\mathbb{E}_{e\sim D}[R_{\pi}(e)]. Together these define the expected-regret self-play game on Δ ⁡ ( ℳ ) × Π \Delta(\mathcal{M})\times\Pi . ##### Pure Nash equilibrium. A pair ( D ∘ , π ∘ ) ∈ Δ ⁡ ( ℳ ) × Π (D^{\circ},\pi^{\circ})\in\Delta(\mathcal{M})\times\Pi is a pure Nash equilibrium of this game if neither player can improve by unilateral deviation: u D ​ ( D ∘ , π ∘ ) ≥ u D ​ ( D , π ∘ ) ∀ D ∈ Δ ⁡ ( ℳ ) , u_{D}(D^{\circ},\pi^{\circ})\geq u_{D}(D,\pi^{\circ})\qquad\forall D\in\Delta(\mathcal{M}), and u A ​ ( D ∘ , π ∘ ) ≥ u A ​ ( D ∘ , π ) ∀ π ∈ Π . u_{A}(D^{\circ},\pi^{\circ})\geq u_{A}(D^{\circ},\pi)\qquad\forall\pi\in\Pi. ##### Optimal values. For each environment, define the optimal unhinted value R ⋆ ​ ( e ) := max π ∈ Π ⁡ R π ​ ( e ) . R^{\star}(e):=\max_{\pi\in\Pi}R_{\pi}(e). ###### Assumption E.1 (Sound generation) . The Environment Designer only samples mathematically valid executable environments: D ∈ Δ ⁡ ( ℳ ) . D\in\Delta(\mathcal{M}). Operationally, this corresponds to syntax checks, executability checks, and, in the tool-use setting, solvability filtering. (Stated for completeness: the game is defined over Δ ⁡ ( ℳ ) \Delta(\mathcal{M}) throughout.) ###### Assumption E.2 (Articulated hints) . Conditioning on the hint attains the optimal unhinted value: for every policy π ∈ Π \pi\in\Pi and every environment e ∈ ℳ e\in\mathcal{M} , R π h ​ ( e ) = R ⋆ ​ ( e ) . R_{\pi}^{h}(e)=R^{\star}(e). In particular R π h ​ ( e ) ≥ R π ​ ( e ) R_{\pi}^{h}(e)\geq R_{\pi}(e) . ###### Assumption E.3 (Internalizability of hinted behavior) . Hinted behavior is attainable without the hint: for every policy π ∈ Π \pi\in\Pi there exists a policy π ′ ∈ Π \pi^{\prime}\in\Pi with R π ′ ​ ( e ) = R π h ​ ( e ) for all ​ e ∈ ℳ . R_{\pi^{\prime}}(e)=R_{\pi}^{h}(e)\qquad\text{for all }e\in\mathcal{M}. ### E.2 Main Results Theorem E.5 shows that every pure Nash equilibrium in this setup drives hint-regret to zero, so that the Reasoning Agent is optimal on every environment in ℳ \mathcal{M} . The ideal hint-regret enables this result, because it vanishes on environments the Reasoning Agent already solves and on unsolvable environments, where the hint cannot help, and is positive only where the Reasoning Agent fails unaided but a hint would close the gap. The Environment Designer therefore profits exactly by targeting this learnable frontier, and at equilibrium no gap can remain anywhere, since the Environment Designer could otherwise deviate to it profitably. This is what distinguishes the regret reward from the plain adversarial-difficulty reward − 𝔼 e ∼ D ​ [ R π ​ ( e ) ] -\mathbb{E}_{e\sim D}[R_{\pi}(e)] , which provides no mechanism restricting the Environment Designer ’s support to useful environments. ###### Lemma E.4 (Hint-regret equals hint-free regret) . Under Assumption E.2 , for any Reasoning Agent policy π ∈ Π \pi\in\Pi and any environment e ∈ ℳ e\in\mathcal{M} , R π h ​ ( e ) − R π ​ ( e ) = R ⋆ ​ ( e ) − R π ​ ( e ) . R_{\pi}^{h}(e)-R_{\pi}(e)=R^{\star}(e)-R_{\pi}(e). In particular the hint-regret is nonnegative, and strictly positive if and only if R π ​ ( e ) < R ⋆ ​ ( e ) R_{\pi}(e)