SPADE: Self-Play in Adaptive Synthetic Executable Environments — part 3/8

Cleaned-text code points [29877, 45254). Bibliography omitted here; source.md includes it.

The settings below differ in what the Environment Designer generates, how it is grounded, how the Reasoning Agent acts, and the regeneration interval k k . Every benchmark is held out from training, and component-level controls such as removing Environment Designer training or grounding are covered by the ablations of Section 7 . Full hyperparameters, per-run adjustments, and evaluation protocol details are in Appendices F.2 and F.3 .

### 5.2 Domain: Games

Environments. The Environment Designer writes each environment as a self-contained Python game with a verifiable reward, and the Reasoning Agent solves it. Each rollout uses 24 24 games, eight for each of three active skills; the six cognitive-skill categories (Mathematical Reasoning, Logical Deduction, Spatial Reasoning, Pattern Recognition, Optimization, and Causal Inference) rotate three at a time, with k = 4 k{=}4 . The generation is grounded in the 15 15 k-document math and science corpus of Section 4.3 . Besides floored hint-based regret, the Environment Designer reward includes the flat-top difficulty anchor of Section 4.1 that pays games whose Reasoning Agent win rate falls in the target band, and every game must pass syntax and execution checks before it enters the pool.

Baselines and evaluation. We retrain two fixed-environment baselines separately for each backbone from its corresponding base checkpoint, using the same 400 400 -iteration budget: Fixed-env RLVE , GRPO on the official RLVE set ( Zeng et al., 2025 ) , and Fixed-env GRPO , which is generated offline by that same Qwen3 backbone and held fixed throughout training. Fixed-env RLVE follows the official RLVE sampling and curriculum settings. RLVE has the higher suite average at every backbone scale. Evaluation spans two distances from the training distribution: procedural reasoning , the four Reasoning-Gym ( Stojanovski et al., 2026 ) categories at hard difficulty, and out-of-distribution (OOD) reasoning and code , AIME 2025/2026 (Avg@32) ( MAA, ) , GPQA-Diamond (accuracy) ( Rein et al., 2023 ) , and LiveCodeBench-v6 (Pass@1) ( Jain et al., 2024 ) . Table 1 reports all eight for every backbone.

### 5.3 Domain: Tool Use

Environments. The Environment Designer instead writes an executable tool-use environment: a set of simulated tools in OpenAI function-calling format, a backend state the tools modify, and three to five natural-language user instructions that arrive one at a time, each with a check on the resulting state. Generation is grounded in the 15 15 k-document code corpus, and tool environments cost more to generate, so we used k = 8 k{=}8 . The Reasoning Agent solves an environment by calling tools over multiple turns and is rewarded only when it completes every user instruction. The privileged hint is a step-by-step plan (look up a record, update it, then reconcile the result), so the Reasoning Agent still has to find the exact arguments by calling the tools. Validation adds two checks beyond the game ones: a deterministic reset gate that exercises every success criterion under several seeds, and an LLM check that every success criterion can be met by some tool (details in Appendix F.2 ). The generation prompt targets the multi-turn function-calling task family of the evaluation suites (schema-defined simulated tools, a backend state, and per-instruction checks); no benchmark tasks or data are shown to the Environment Designer .

Baselines and evaluation. We compare SPADE with four synthetic-environment systems: AgentScaler ( Fang et al., 2025a ) , Agent-World ( Dong et al., 2026 ) , Agent World Model (AWM) ( Wang et al., 2026b ) , and EnvScaler ( Song et al., 2026 ) . Their results are transcribed from the cited papers and differ from ours in training data and budget, in some cases base model, and in evaluation protocol (Appendix F.3.1 ). Evaluation uses BFCL v4 multi-turn ( Patil et al., 2025 ) , τ 2 \tau^{2} -bench ( Barres et al., 2025 ) , and ACEBench-Agent (the Agent category of ACEBench-en) ( Chen et al., 2025a ) , reported per domain in Table 2 .

## 6 Experimental Results

We evaluate SPADE on two settings, game and tool-use environment design, across the three Qwen3 backbones of Section 5.1 . Section 6.1 reports held-out benchmark performance against the fixed-environment baselines, and Section 6.2 examines what the Environment Designer and Reasoning Agent produce over training.

### 6.1 Quantitative Analysis

Synthetic game environments improve science, code, and procedural reasoning. At 30B-A3B, SPADE reaches a suite average of 58.3 58.3 : + 8.1 +8.1 over base and + 5.3 +5.3 over the strongest fixed-environment baseline (Fixed-env RLVE), a margin that grows with model size (Table 1 ). The Environment Designer trains on synthetic games, never on any held-out task, yet the gains transfer to science, code, and procedural reasoning and hold late in the 400 400 -step run (Figure 6 ), while competition math is preserved. They concentrate on procedural reasoning, where every cognitive-skill category improves: the generated games exercise each skill through many problem structures rather than a fixed task set.

Synthetic tool-use environments improve multi-step interaction. The same recipe applied to tool-use environment design improves every backbone (Table 2 ), and at 30B-A3B the size of the gain tracks how closely a benchmark’s task structure matches the generated environments: ACEBench-Agent gains most ( + 13.9 +13.9 ), and its stateful, multi-step tasks mirror the generated pattern of a database, a tool schema, and a multi-call goal. BFCL v4 multi-turn follows ( + 5.7 +5.7 , and + 10.3 +10.3 at 4B), then τ 2 \tau^{2} -bench ( + 3.6 +3.6 ). SPADE at 30B-A3B leads the dedicated data-synthesis systems on both BFCL v4 multi-turn and ACEBench-Agent, so structural training signal transfers where domain-specific data collection does not reach.

### 6.2 Qualitative Analysis

The Environment Designer keeps supplying environments the Reasoning Agent can learn from. We examine what each role produces over the main 30B games run. Figure 8 tracks the learnable share of each rollout, the fraction of generated environments on which the Reasoning Agent wins between 20 % 20\% and 80 % 80\% of the time. Full SPADE raises this share over training, reaching roughly a third late in the 400-step run; the corpus, the memory, and Environment Designer training together sustain that supply. Figure 8 traces the diversity of that supply to the corpus: corpus grounding keeps the environments diverse (Vendi/ n n 0.68 0.68 , versus 0.04 0.04 without it), and the control without Environment Designer training or memory, but with the corpus, retains full diversity ( 0.70 0.70 , Appendix I.1.2 ). Corpus grounding provides the breadth, while the full Environment Designer training configuration sharpens the difficulty. Ivison (2026) reaches the same conclusion from the failure side: an ungrounded proposer collapses onto a handful of near-identical programs, explicit diversity rewards get hacked once they are optimized against, and conditioning on an external corpus is the intervention that holds longest. Figure 8 shows that corpus grounding sustains diversity, whereas the ungrounded run collapses.

Environment Designer training makes environments harder in measurable, code-level ways. The generation instructions are fixed across all 400 steps (the same template asking for hidden state, multi-turn structure, and partial reward), while the grounding document is resampled from the corpus each step with no trend over training, so any systematic change in the generated environments traces to Environment Designer training rather than to a prompt schedule. Two measurements move together. First, within Physics, the share of environments whose opening observation prints the governing formula falls from 25 % 25\% to 5 % 5\% over the 473 473 Physics environments of the canonical run (Figure 9 ). Second, rewards become more finely graded, from 3.7 3.7 to 5.8 5.8 distinct levels per environment, of which 2.2 → 4.0 2.2\to 4.0 are strictly partial (Appendix I.1.1 ). The no-corpus run (w/o corpus grounding in Table 3 ) shows none of this: over steps 290–312 it emits the same rotating-maze task 41 times in a row.

The Reasoning Agent learns to act on evidence rather than derive in advance. Figure 10 shows the Reasoning Agent ’s side of the same run. At step 0 it reasons entirely up front and cannot recover once the interface rejects its answer; by step 200 it tests short hypotheses and revises them as results return; by step 300 it gathers evidence first and derives once, when the evidence is enough. The shift appears only where the task rewards inference: on procedural environments the same policy acts in short commands (late-step median 8 8 tokens). The model still derives at length when the task calls for it, and the late-checkpoint benchmark gains (Table 1 ) confirm it keeps the ability.

## 7 Ablations

We ablate two design choices for the Environment Designer : whether it adapts during training (Section 7.1 ), and how it is rewarded (Section 7.2 ). Table 3 covers the ablation for training settings and environment sources. The first two settings remove corpus grounding and the environment memory, one at a time. The remaining two variants remove online Environment Designer training in different ways: one continues self-generation but removes both Environment Designer training and the environment memory; the other trains the Reasoning Agent on a static pool generated offline by GPT-5.5, with corpus grounding. All variants use games on Qwen3-30B-A3B-Instruct-2507 and are evaluated on the same eight-benchmark suite as the main experiments. Each variant reports its best checkpoint on the suite average; full eval curves are in Figure 11 , and per-ablation breakdowns in Appendix G .

### 7.1 Ablation: Environment Designer Adaptation

As shown in Table 3 and Figure 11 , the full SPADE run is strongest across settings. Removing both Environment Designer training and memory makes the model worse than no training at all, with the eight-benchmark average falling to 40.5 40.5 , 9.7 9.7 points below the untrained base. The static GPT-5.5 pool, generated offline with corpus grounding but no environment memory, does better, beating the base ( 53.0 53.0 versus 50.2 50.2 ), but recovers only about 35 % 35\% of SPADE ’s + 8.1 +8.1 gain and does not improve code (LiveCodeBench-v6 42.6 42.6 versus 43.2 43.2 ). The partial and non-adaptive variants peak early and then fade (the no-memory and no-corpus runs near step 111, the GPT-5.5 pool near step 175), while full SPADE stays strongest late in training (Figure 11 ). Each control changes more than one factor: the self-generation control removes both Environment Designer training and memory, while the GPT-5.5 control changes the generator and uses an offline, memory-free pool. Together they show that the full co-adaptive setup in SPADE wins.

### 7.2 Ablation: Environment Designer Reward

SPADE ’s Environment Designer reward is built on hint-based regret (Eq. 3 ), blended with the difficulty anchor of Section 4.1 . The alternative below is cheaper: it reuses the Reasoning Agent rollouts already collected and needs no hinted replay.

EMA-based learning potential. This reward design ( Kanitscheider et al., 2021 ; Zhang et al., 2023 ) scores an environment by how far the Reasoning Agent ’s success on it departs from its skill’s recent average. Environments are grouped by skill s s . Let r ¯ A ​ ( e ) \bar{r}_{A}(e) be the Reasoning Agent ’s average return on environment e e (Eq. 3 ), and r ¯ A , t ​ ( s ) \bar{r}_{A,t}(s) its mean over the skill’s environments in scoring round t t . Two moving averages of r ¯ A , t ​ ( s ) \bar{r}_{A,t}(s) are tracked at rates γ 1 > γ 2 \gamma_{1}>\gamma_{2} : μ t γ ​ ( s ) = ( 1 − γ ) ​ μ t − 1 γ ​ ( s ) + γ ​ r ¯ A , t ​ ( s ) , μ fast ≡ μ γ 1 , μ slow ≡ μ γ 2 . \mu^{\gamma}_{t}(s)\;=\;(1-\gamma)\,\mu^{\gamma}_{t-1}(s)\;+\;\gamma\,\bar{r}_{A,t}(s),\qquad\mu_{\text{fast}}\equiv\mu^{\gamma_{1}},\quad\mu_{\text{slow}}\equiv\mu^{\gamma_{2}}. (4) Here γ \gamma weights the new observation, so μ fast \mu_{\text{fast}} adapts faster than μ slow \mu_{\text{slow}} , and μ γ \mu^{\gamma} has half-life log ⁡ ( 0.5 ) / log ⁡ ( 1 − γ ) \log(0.5)/\log(1-\gamma) rounds. The reward uses μ slow \mu_{\text{slow}} as the skill baseline: ρ ⁡ ( e ) \displaystyle\rho(e)\; = | r ¯ A ​ ( e ) − μ slow , t ​ ( s ) | , \displaystyle=\;\bigl|\,\bar{r}_{A}(e)-\mu_{\text{slow},t}(s)\,\bigr|, (5) r D LP ​ ( e ) \displaystyle r_{D}^{\text{LP}}(e)\; = ρ ⁡ ( e ) − 1 | ℰ s | ​ ∑ e ′ ∈ ℰ s ρ ⁡ ( e ′ ) , \displaystyle=\;\rho(e)-\tfrac{1}{|\mathcal{E}_{s}|}\textstyle\sum_{e^{\prime}\in\mathcal{E}_{s}}\rho(e^{\prime}), (6) where ℰ s \mathcal{E}_{s} is the round’s environment set for skill s s .

Two consequences follow. The deviation is unsigned, so an environment the Reasoning Agent always solves can score as highly as one it never solves when the slow mean sits mid-range, though neither gives gradient; only a variance-style bonus such as r ¯ A ​ ( e ) ​ ( 1 − r ¯ A ​ ( e ) ) \bar{r}_{A}(e)\bigl(1-\bar{r}_{A}(e)\bigr) would favour mixed outcomes. And μ fast \mu_{\text{fast}} is tracked but unused: the classical gap | μ fast − μ slow | |\mu_{\text{fast}}-\mu_{\text{slow}}| is logged only as a diagnostic. The signal also needs a per-skill history before it is meaningful, and cannot separate deviation caused by environment design from Reasoning Agent drift or sampling noise.

Results. Figure 12 shows the trajectories for the two trained- Environment Designer rewards and the control without Environment Designer training or memory on the six-benchmark trajectory average, while Table 6 gives their selected checkpoints on the full eight-benchmark suite. Hint-based regret lifts the eight-benchmark average from 50.2 50.2 to 58.3 58.3 ( + 8.1 +8.1 ); EMA-based learning potential reaches around 70 % 70\% of that gain ( + 5.7 +5.7 , to 55.9 55.9 ) and climbs more slowly, the two signals separating after the first ∼ 50 {\sim}50 steps. Both stay far clear of the control without Environment Designer training or memory, whose selected checkpoint is 9.7 9.7 points below the untrained model. The comparison between the two trained- Environment Designer variants isolates the reward choice, the regret-based blend versus the standalone learning-potential signal; the third comparison does not isolate Environment Designer training from memory. The form of Eq. 6 helps explain the remaining reward gap: an unsigned deviation from a per-skill running mean needs history before it means anything, and even then scores mastered and hopeless environments alike, so it finds the frontier later and less sharply than a hint gap measured on the current policy. Per-benchmark numbers are in Table 6 .

## 8 Scaling Results

Scaling model size. Under a shared training recipe (Section 5.1 ), the average gain over each backbone’s base grows from + 5.2 +5.2 at 4B and + 5.7 +5.7 at 8B to + 8.1 +8.1 at 30B-A3B, while Fixed-env GRPO stays near + 1.2 +1.2 at every size (Figure 13 ; Table 1 ). Fixed-env GRPO supplies a static training signal, whereas the Environment Designer keeps adapting the curriculum to the Reasoning Agent ; the widening gap is consistent with larger models benefiting more from that adaptivity.