SPADE: Self-Play in Adaptive Synthetic Executable Environments — part 2/8

Cleaned-text code points [14309, 29877). Bibliography omitted here; source.md includes it.

Environment Synthesis and Scaling Following the shift from single-turn reinforcement learning with verifiable rewards (RLVR) ( OpenAI, 2024 ; DeepSeek-AI, 2025 ) to multi-turn agentic RL ( Wang et al., 2025c ; Zhang et al., 2025c ) , the env-pool bottleneck has motivated work that synthesizes training environments for LLM agents at scales unreachable by hand-curation. Recent systems generate agentic worlds and toolsets ( Wang et al., 2026b ; Tu et al., 2026 ; Dong et al., 2026 ) , terminal and computer-use sandboxes ( Zhu et al., 2026 ; Gandhi et al., 2026 ; Fan et al., 2026 ; Pi et al., 2026 ; Xue et al., 2026a ; Xue et al., 2026b ; Zhang et al., 2026e ) , and dynamic RL pipelines ( Wang et al., 2026a ; Guo et al., 2025 ; Shi et al., 2026 ) ; AgentScaler ( Fang et al., 2025a ) clusters thousands of real APIs into per-domain Python tool environments and trains agents via filtered supervised fine-tuning (SFT), and Eurekaverse ( Liang et al., 2024 ) provides a robotics analogue. Complementary task-level synthesis grounds generation in agent exploration ( Ramrakhya et al., 2025 ) , converts unverifiable text into RLVR data ( Lu et al., 2026 ) , or expands seed task spaces ( Jiang et al., 2025b ; Chen et al., 2025e ; Li et al., 2025 ; Lu et al., 2025 ) . Scaling studies show that distributions of verifiable environments lift generalization: WebScale-RL ( Cen et al., 2025 ) mines RL tasks from web pretraining data, AgentRL ( Zhang et al., 2025a ) unifies multi-turn multi-task training, and RLVE ( Zeng et al., 2025 ) hand-engineers 400 verifiable environments with adaptive difficulty levels. Curriculum selection over fixed pools provides difficulty adaptation without changing the underlying environments ( Chen et al., 2025d ; Xu et al., 2026a ) . Across these systems the environment generator is typically frozen, hand-engineered, or trained on a separate signal. SPADE instead trains the Environment Designer online, together with the Reasoning Agent , via RL with hint-based regret, produces full multi-turn MDP environments as executable Python rather than tasks with sparse terminal rewards, and co-evolves the environment distribution with the Reasoning Agent ’s capability frontier in a single training loop.

## 3 Preliminaries

SPADE builds on two standard components, which we review before presenting the method: the Markov decision process with a Gym-style reset() / step() interface, used to represent environments, and reinforcement learning from verifiable rewards optimized with GRPO, used to train the policy.

### 3.1 Markov Decision Processes and Gym-Style Environments

A Markov Decision Process (MDP) is a tuple ( 𝒮 , 𝒜 , T , R , ρ 0 ) (\mathcal{S},\mathcal{A},T,R,\rho_{0}) with state space 𝒮 \mathcal{S} , action space 𝒜 \mathcal{A} , transition function T ⁡ ( s ′ ∣ s , a ) T(s^{\prime}\mid s,a) , reward function R ⁡ ( s , a ) R(s,a) , and initial state distribution ρ 0 \rho_{0} . The standard programmatic interface for an MDP, popularized by OpenAI Gym ( Brockman et al., 2016 ) and standardized in its Gymnasium fork ( Towers et al., 2026 ) , exposes two functions: reset() returns an initial observation s 0 ∼ ρ 0 s_{0}\sim\rho_{0} (with an info dictionary), and step ( a a ) advances the state via T T and returns ( s ′ , r , terminated , truncated , info ) (s^{\prime},r,\text{terminated},\text{truncated},\text{info}) , separating natural termination from time-limit truncation. Our generated environments are fully observed: the observation returned at each step is the state itself. Listing 3 shows a minimal instantiation.

### 3.2 RLVR and GRPO

We train the policy π θ \pi_{\theta} , an LLM with parameters θ \theta , via reinforcement learning with verifiable rewards (RLVR) using Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) . For each input prompt x x , GRPO samples a group of G G responses { y 1 , … , y G } ∼ π θ ( ⋅ ∣ x ) \{y_{1},\ldots,y_{G}\}\sim\pi_{\theta}(\cdot\mid x) and computes group-normalized advantages: A ^ i = r i − mean ​ ( { r j } j = 1 G ) std ​ ( { r j } j = 1 G ) \hat{A}^{i}=\frac{r^{i}-\text{mean}(\{r^{j}\}_{j=1}^{G})}{\text{std}(\{r^{j}\}_{j=1}^{G})} (1) The policy is updated via clipped policy gradient with KL regularization: ℒ ( θ ) = − 1 G ∑ i = 1 G min ( π θ ​ ( y i ∣ x ) π old ​ ( y i ∣ x ) A ^ i , clip ( π θ ​ ( y i ∣ x ) π old ​ ( y i ∣ x ) , 1 − ε low , 1 + ε high ) A ^ i ) + β KL ⋅ KL [ π θ ∥ π ref ] \mathcal{L}(\theta)=-\frac{1}{G}\sum_{i=1}^{G}\min\left(\frac{\pi_{\theta}(y_{i}\mid x)}{\pi_{\text{old}}(y_{i}\mid x)}\hat{A}^{i},\;\text{clip}\!\left(\frac{\pi_{\theta}(y_{i}\mid x)}{\pi_{\text{old}}(y_{i}\mid x)},1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}}\right)\hat{A}^{i}\right)+\beta_{\text{KL}}\cdot\text{KL}[\pi_{\theta}\|\pi_{\text{ref}}] (2)

## 4 SPADE : Self-Play in Adaptive Synthetic Executable Environments

SPADE is an end-to-end self-play framework where a single LLM, π θ \pi_{\theta} , alternates between generating executable training environments and learning to solve them (Figure 4 ; Algorithm 1 ). Its three components create an adaptive curriculum: § 4.1 details how dual-role self-play couples environment generation with Reasoning Agent (RA) learning; § 4.2 explains how hint-based regret steers the Environment Designer (ED) toward environments at the Reasoning Agent ’s learning frontier; and § 4.3 outlines the pipeline that grounds and validates these executable environments.

### 4.1 Dual-Role Self-Play and Code-as-Environment

SPADE formulates adaptive environment self-play as a game where a single model π θ \pi_{\theta} acts in two roles via role-specific system prompts: designing environments ( role = D \text{role}{=}D ) or solving them ( role = A \text{role}{=}A ). Let ℰ \mathcal{E} denote the space of all valid environments. In the Environment Designer role ( π D \pi_{D} ), π θ \pi_{\theta} produces an executable environment e ∈ ℰ e\in\mathcal{E} as a Python program implementing the Gym-style reset() / step() API (Section 3.1 ; a minimal example is shown in Appendix J.3.1 ), where the transition function T ⁡ ( s ′ ∣ s , a ) T(s^{\prime}\mid s,a) and reward function R ⁡ ( s , a ) R(s,a) are both encoded in the step() implementation. The Environment Designer also emits a privileged hint for each environment. A hint h h is task-relevant information that the Environment Designer attaches to an environment (for example, a partial solution sketch or a key structural observation); revealing h h to the Reasoning Agent makes the environment easier to solve, and the gap in Reasoning Agent return with versus without h h defines the Environment Designer ’s hint-based regret reward (Section 4.2 ).

In the Reasoning Agent role ( π A \pi_{A} ), π θ \pi_{\theta} interacts with e e via sequential actions, receiving observations and rewards. This code-as-environment representation unifies single-turn settings ( reset() → \rightarrow step(answer) → \rightarrow terminal reward) and multi-turn agentic settings ( reset() → \rightarrow step() → ⋯ → \rightarrow\cdots\rightarrow done ) under a single interface and training pipeline. This representation provides the vast task space that AI-GAs ( Clune, 2019 ) argue is necessary for open-ended self-improvement: any computable MDP can be expressed as a Python program, so the Environment Designer can express any such environment in code rather than being confined to a hand-designed environment parameterization. Both roles share parameters θ \theta , so updates for either role affect the same policy. Each self-play cycle first collects Environment Designer trajectories, and then Reasoning Agent trajectories from those generated environments. The Environment Designer receives hint-based regret (Section 4.2 ), whereas the Reasoning Agent receives task correctness from the environment’s reward function. Each candidate environment is validated for syntactic correctness and executability before entering the training pool; validation details are in Appendix F.2 .

Stabilizing joint two-role training. Optimizing a single policy for two coupled objectives requires several stabilizing techniques ( Liu et al., 2026a ) . Following SPIRAL ( Liu et al., 2025a ) , we normalize advantages independently for each role. We standardize Reasoning Agent returns within each environment’s rollouts and mean-center Environment Designer rewards within each skill. To ensure both roles contribute comparably, we upweight the less frequent Environment Designer trajectories. Finally, we delay the Environment Designer update by k k rollouts, where k k is the number of training steps per environment set, so the difficulty anchor can score each environment’s Reasoning Agent win rate over the full training window; the regret component is computed at generation time. Since this delay makes the Environment Designer objective off-policy, we correct the gradient using truncated importance sampling ( Yao et al., 2025 ) . We use an asymmetric clipping range ( ε low = 0.2 \varepsilon_{\text{low}}{=}0.2 , ε high = 0.28 \varepsilon_{\text{high}}{=}0.28 ) to preserve exploration and floor the raw regret at zero. The deployed Environment Designer reward blends this floored regret, normalized to [ 0 , 1 ] [0,1] by a fixed scale, (weight 0.4 0.4 ) with a flat-top difficulty anchor that pays environments whose Reasoning Agent win rate falls in a target band and decays linearly outside it (weight 0.6 0.6 , band [ 0.4 , 0.6 ] [0.4,0.6] ): the anchor regulates difficulty, and floored regret selects the most teachable environments within the band. Full hyperparameters appear in Appendix F.2 .

### 4.2 Hint-Based Regret Reward

The Environment Designer ’s reward for producing environment e e is: r D ​ ( e ) = r ¯ A ​ ( e ∣ h ) − r ¯ A ​ ( e ) , r_{D}(e)=\bar{r}_{A}(e\mid h)-\bar{r}_{A}(e), (3)

where r ¯ A ​ ( e ∣ h ) = 1 G ​ ∑ i = 1 G r A ​ ( y i ′ ∣ e , h ) \bar{r}_{A}(e\mid h)=\frac{1}{G}\sum_{i=1}^{G}r_{A}(y^{\prime}_{i}\mid e,h) is the Reasoning Agent ’s average return over G G fresh rollouts y i ′ y^{\prime}_{i} sampled with the privileged hint h h in context, and r ¯ A ​ ( e ) \bar{r}_{A}(e) is the corresponding average over G G rollouts without it. Three regimes emerge: high regret indicates an environment at the learning frontier (solvable with hints but not without); low regret where the Reasoning Agent succeeds with and without the hint indicates mastery; and low regret with low returns even with the hint indicates an intractable environment. This implements the minimax regret objective of PAIRED ( Dennis et al., 2020 ) without a separate antagonist: the hint-equipped Reasoning Agent serves as the upper-bound policy. For a policy that best-responds to the hint, extra information cannot reduce expected return, so regret is non-negative at the optimum; for the current policy, expected regret can dip below zero when hints mislead it, which we observe for the smaller backbones (Figure 13 ). We formalize this intuition and show that, under idealized assumptions, every pure Nash equilibrium yields hint-free optimality on every valid environment (Appendix E ). We compare hint-based regret against an EMA-based learning-potential signal ( Kanitscheider et al., 2021 ; Zhang et al., 2023 ) in the Environment Designer -reward ablation (Section 7.2 ).

Hint generation. For each environment e e , the Environment Designer (same LLM, different prompt) emits one task-specific hint of a few sentences. The prompt asks for the key insight or strategy and the expected answer format, but forbids revealing the exact answer; its output is unlabeled hint text. Figure 5 shows two high-regret examples. Each hint is shown beside the task it conditions and one logged Reasoning Agent rollout from each prompt condition. The paired trajectories expose the behavioral source of the return gap: the hint hands the fiber-task agent the probe pattern almost outright, and narrows the audio-task agent’s search to a parameter range it would otherwise reach only after repeated failed attempts.

### 4.3 Environment Design

Corpus grounding. A generator conditioned only on its own output has no source of novelty outside its own weights, so it narrows toward the patterns it already favors and mode-collapses, the “invisible leash” ( Chae et al., 2025 ; Zhang et al., 2026c ; Jiang et al., 2025a ) . We treat an external corpus as the mechanism that lengthens that leash: every round the Environment Designer conditions on freshly sampled human corpus. In the games setting these are 10 10 k mathematics and 5 5 k science documents drawn from DCLM ( Li et al., 2024 ) and MegaScience ( Fan et al., 2025 ) , spanning web pages, physics forums, and university-level scientific textbooks; in the tool-use setting, 15 15 k documents from the Nemotron pretraining code corpus ( NVIDIA, 2026 ) , which supplies algorithm implementations and API documentation. SPICE ( Liu et al., 2025b ) established corpus grounding for task generation, mining documents for reasoning questions; SPADE carries the principle to environment generation, where a sampled passage seeds an executable MDP rather than a question-answer pair. Section 6.2 (Figure 8 ) shows how corpus grounding sustains environment diversity.

Environment memory. The corpus supplies breadth from outside the loop; the cross-episode memory keeps the Environment Designer from re-posing what the Reasoning Agent has already mastered ( Schmidhuber, 2013 ) . A buffer of previously generated environments, annotated with regret scores and skill tags, gives the Environment Designer high-regret seeds to vary, so each round starts from what the Reasoning Agent currently finds hard rather than from scratch. Retaining and reusing past experience this way is the mechanism agentic-memory systems rely on to keep improving without further gradient updates ( Zhang et al., 2026d ; Xiong et al., 2026 ) ; here it acts on the design side, holding generated difficulty at the Reasoning Agent ’s frontier as that frontier moves. The two inputs work on different axes, the corpus on what environments are about and the memory on how hard they are, and Table 3 separates their contributions.

Environment pool management, lifecycle, and quality control details are in Appendix F.2 .

## 5 Experimental Setup

### 5.1 SPADE Training Recipe

We train three Qwen3 backbones: Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507, the last of which is our primary model. The 4B and 30B runs use instruct models; the 8B run enables thinking for both the Environment Designer and the Reasoning Agent . We train with GRPO ( Shao et al., 2024 ) for 400 400 rollouts of 24 24 environments each. The Environment Designer regenerates the environment set every k k rollouts and its update is delayed by the same k k (Section 4.1 ), so k k sets how long the Reasoning Agent trains on a fixed set before the curriculum moves. The Reasoning Agent plays every environment 16 × k 16\times k times without the hint and 16 16 times with it; the regret subtracts hint and no-hint averages measured at the same regeneration step ( 16 16 plays each). The Environment Designer writes the privileged hint itself, with no external model, and each round it also draws high-regret seeds from a memory of past environments. Rewards are normalized per environment, and the stabilizers described earlier (per-role advantages, the delayed Environment Designer update with truncated importance sampling, and the regret floor) are unchanged across settings. Our RL backbone is slime ( Zhu et al., 2025 ) .