SPADE: Self-Play in Adaptive Synthetic Executable Environments — part 4/8
Cleaned-text code points [45254, 60799). Bibliography omitted here; source.md includes it.
Scaling curriculum diversity. The cognitive-skill curriculum is the set of skill categories the Environment Designer targets when it generates environments: the six listed in Section 5.2 , three active per regeneration in round-robin. We compare the full six-skill curriculum against a two-skill version, with everything else held fixed (Figure 14 ; per-benchmark panels in Figure 15 , Appendix G ). The two-skill run also improves, but it captures only about half of the Reasoning-Gym gains and much smaller GPQA-Diamond and LiveCodeBench-v6 gains (best checkpoint on the eight-benchmark suite of Table 1 : 53.7 vs. 58.3). The gains grow with the diversity of the curriculum, not with any single game family.
## 9 Discussion
SPADE ’s results support two conclusions. First, training the Environment Designer to produce environments adapted to the Reasoning Agent ’s current ability outperforms training on fixed environment sources: both the main Fixed-env GRPO pools generated once offline by the corresponding Qwen3 backbones and held fixed throughout training (Table 1 , Section 6.1 ), and the offline pool generated by the stronger GPT-5.5 model in the ablation (Table 3 , Section 7 ). The widening gap over Fixed-env GRPO with model scale is consistent with an adaptivity advantage. Second, one code-as-environment interface spans two very different settings. In the games setting, reasoning skills the Reasoning Agent develops (planning, constraint satisfaction, strategic thinking) generalize to held-out mathematics, science, and code, well beyond the game format they were learned in. In the tool-use setting, the same recipe lifts multi-step agentic benchmarks most where interaction matters, by + 13.9 +13.9 on ACEBench-Agent and + 5.7 +5.7 on BFCL v4 multi-turn at 30B-A3B. The same learnable environment-design role is effective in both settings.
Limitations. (a) Complexity bounded by scale and the invisible leash ( Chae et al., 2025 ) . The Environment Designer cannot produce environments more complex than its base model can express in context, so reachable environment complexity grows with model scale and generation budget. (b) A human-designed optimizer. Both roles are updated by a fixed, human-authored RL algorithm (GRPO); SPADE does not modify its own learning rule. (c) No formal optimality, and fixed-task evaluation. Hint-based regret is motivated by PAIRED ( Dennis et al., 2020 ) but not proven to yield an optimal curriculum, and current benchmarks measure fixed-task performance rather than open-ended reasoning growth.
Future directions. We improve the Environment Designer with gradient updates, but co-adaptation between the Environment Designer and Reasoning Agent might also come from an Environment Designer that improves without weight updates, accumulating and refining design strategies from past attempts in context; whether learned weights or in-context evolution makes the better designer, and at what scale, is an open question. SPADE also automates one stage of post-training; combining adaptive environment design with systems that automate the remaining stages, from data curation through the learning rule itself, could extend co-adaptive self-play across the whole training pipeline.
## 10 Conclusion
We presented SPADE , a framework that makes environment design a learnable component of LLM post-training through self-play. Three contributions enable this: (1) a hint-based regret reward that trains the Environment Designer to produce environments at the Reasoning Agent ’s learning frontier, grounded in minimax regret theory; (2) environment design anchored in a pretraining corpus and an environment memory, with a code-as-environment interface that unifies single-turn and multi-turn settings; and (3) a practical recipe at 30B+ scale on Qwen3 models. In the games setting, SPADE raises held-out reasoning and code benchmarks by + 8.1 +8.1 average points over base at 30B-A3B ( + 5.2 +5.2 and + 5.7 +5.7 at 4B and 8B); these gains remain at late checkpoints of the 400 400 -step run, and the margin over the strongest fixed-environment baseline grows with scale. The same recipe lifts multi-step tool-use benchmarks by + 13.9 +13.9 on ACEBench-Agent and + 5.7 +5.7 on BFCL v4 multi-turn.
SPADE shows that a single model can design its own training environments and improve from them, a step from fixed benchmarks toward open-ended, continual self-improvement.
## Acknowledgments
This research was supported by the UW-Amazon Science Gift Hub, UW-Tsukuba Amazon NVIDIA Cross Pacific AI Initiative (XPAI), Sony Research Award, Modal Research Grants, Tinker Research Grants, Character.AI, DoorDash, Open Philanthropy, Coefficient Giving, Toyota Research Institute, and the Schmidt AI2050 Fellows program. This material is based upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number(s): FA8650-23-C-7316. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL or DARPA.
## Appendix E Theoretical Analysis [back to contents]
This section formally analyzes the incentive structure of the hint-based regret reward (hereafter hint-regret) introduced in Section 4.2 . We model the interaction between the Environment Designer and Reasoning Agent as a two-player game: the Environment Designer selects an environment distribution, while the Reasoning Agent selects a policy. Since LLM inference is stochastic, all payoffs are defined through expected verifier returns.
The main result is an equilibrium characterization of the idealized game. If positive hint-regret remains anywhere, then the current unhinted Reasoning Agent is suboptimal there, and the Environment Designer can profitably concentrate on it. Consequently, at any pure Nash equilibrium, the Environment Designer ’s expected regret is zero, the Reasoning Agent is hint-free optimal on every environment in ℳ \mathcal{M} , and privileged hints become vacuous.
### E.1 Setup and Assumptions
##### Environment class.
Let 𝒰 \mathcal{U} be a finite universe of candidate environments. The set of mathematically valid executable environments is denoted by ℳ ⊆ 𝒰 \mathcal{M}\subseteq\mathcal{U} . These are environments whose specifications are internally consistent and whose attempted solutions can be evaluated by an external verifier. In the implementation, ℳ \mathcal{M} is approximated by syntax checks, executability checks, and, in the tool-use setting, solvability filtering.
##### Distribution notation.
For any finite set 𝒳 \mathcal{X} , let Δ ( 𝒳 ) \Delta(\mathcal{X}) denote the set of probability distributions over 𝒳 \mathcal{X} . If D ∈ Δ ( ℳ ) D\in\Delta(\mathcal{M}) (the Environment Designer ’s environment distribution; we reuse the letter D D for this distribution, while the subscript in u D u_{D} below labels the Environment Designer role) and ℬ ⊆ ℳ \mathcal{B}\subseteq\mathcal{M} , write D ( ℬ ) := ∑ e ∈ ℬ D ( e ) . D(\mathcal{B}):=\sum_{e\in\mathcal{B}}D(e). A property holds for D D -almost every environment if the set on which it fails has D D -mass zero.
##### Inference protocol.
Fix an inference budget B inf B_{\text{inf}} . This budget includes all evaluation-time choices that affect the set of possible Reasoning Agent trajectories: model architecture, context length, prompt format, maximum generation length, sampling rule, temperature, number of rollouts, tool access, memory access, and verifier. Let 𝒴 B inf \mathcal{Y}_{B_{\text{inf}}} denote the finite trajectory space induced by this budget.
##### Fine-tuning budget.
Fix a fine-tuning budget F F . This budget includes all training-time choices that determine which policies the procedure can produce: optimizer, objective, number of updates, and regularization. Let Π \Pi denote the set of policies attainable under this budget; we take Π \Pi to be finite, so maxima over Π \Pi are attained.
##### Returns with and without hints.
For each environment e ∈ ℳ e\in\mathcal{M} , let h ( e ) h(e) denote the privileged hint emitted by the Environment Designer ; hints are modeled as a fixed map e ↦ h ( e ) e\mapsto h(e) , so strategic hint choice by the Environment Designer is outside this idealization. A Reasoning Agent policy π ∈ Π \pi\in\Pi induces a distribution over trajectories in 𝒴 B inf \mathcal{Y}_{B_{\text{inf}}} given either the unhinted input e e or the hinted input ( e , h ( e ) ) (e,h(e)) . Let r ( e , y ) ∈ [ 0 , 1 ] r(e,y)\in[0,1] denote the verifier return of trajectory y ∈ 𝒴 B inf y\in\mathcal{Y}_{B_{\text{inf}}} on environment e e . Returns are normalized to [ 0 , 1 ] [0,1] for the analysis; training uses a shaped reward in [ − 1 , 1 ] [-1,1] (Algorithm 1 ). Here r ( e , y ) r(e,y) is the per-trajectory return written r A ( y ∣ e ) r_{A}(y\mid e) in Section 4 , and R π ( e ) R_{\pi}(e) , R π h ( e ) R_{\pi}^{h}(e) below are the population analogues of the empirical means r ¯ A ( e ) \bar{r}_{A}(e) and r ¯ A ( e ∣ h ) \bar{r}_{A}(e\mid h) .
The unhinted expected return of policy π \pi on environment e e is R π ( e ) := 𝔼 y ∼ π ( ⋅ ∣ e ) [ r ( e , y ) ] , R_{\pi}(e):=\mathbb{E}_{y\sim\pi(\cdot\mid e)}[r(e,y)], and the hinted expected return is R π h ( e ) := 𝔼 y ∼ π ( ⋅ ∣ e , h ( e ) ) [ r ( e , y ) ] . R_{\pi}^{h}(e):=\mathbb{E}_{y\sim\pi(\cdot\mid e,h(e))}[r(e,y)].
##### Game payoffs.
The Environment Designer chooses a distribution D ∈ Δ ( ℳ ) D\in\Delta(\mathcal{M}) . Its payoff is the expected hint-based regret u D ( D , π ) := 𝔼 e ∼ D [ R π h ( e ) − R π ( e ) ] . u_{D}(D,\pi):=\mathbb{E}_{e\sim D}\left[R_{\pi}^{h}(e)-R_{\pi}(e)\right]. The Reasoning Agent ’s payoff is its unhinted expected return u A ( D , π ) := 𝔼 e ∼ D [ R π ( e ) ] . u_{A}(D,\pi):=\mathbb{E}_{e\sim D}[R_{\pi}(e)]. Together these define the expected-regret self-play game on Δ ( ℳ ) × Π \Delta(\mathcal{M})\times\Pi .
##### Pure Nash equilibrium.
A pair ( D ∘ , π ∘ ) ∈ Δ ( ℳ ) × Π (D^{\circ},\pi^{\circ})\in\Delta(\mathcal{M})\times\Pi is a pure Nash equilibrium of this game if neither player can improve by unilateral deviation: u D ( D ∘ , π ∘ ) ≥ u D ( D , π ∘ ) ∀ D ∈ Δ ( ℳ ) , u_{D}(D^{\circ},\pi^{\circ})\geq u_{D}(D,\pi^{\circ})\qquad\forall D\in\Delta(\mathcal{M}), and u A ( D ∘ , π ∘ ) ≥ u A ( D ∘ , π ) ∀ π ∈ Π . u_{A}(D^{\circ},\pi^{\circ})\geq u_{A}(D^{\circ},\pi)\qquad\forall\pi\in\Pi.
##### Optimal values.
For each environment, define the optimal unhinted value R ⋆ ( e ) := max π ∈ Π R π ( e ) . R^{\star}(e):=\max_{\pi\in\Pi}R_{\pi}(e).
###### Assumption E.1 (Sound generation) .
The Environment Designer only samples mathematically valid executable environments: D ∈ Δ ( ℳ ) . D\in\Delta(\mathcal{M}). Operationally, this corresponds to syntax checks, executability checks, and, in the tool-use setting, solvability filtering. (Stated for completeness: the game is defined over Δ ( ℳ ) \Delta(\mathcal{M}) throughout.)
###### Assumption E.2 (Articulated hints) .
Conditioning on the hint attains the optimal unhinted value: for every policy π ∈ Π \pi\in\Pi and every environment e ∈ ℳ e\in\mathcal{M} , R π h ( e ) = R ⋆ ( e ) . R_{\pi}^{h}(e)=R^{\star}(e). In particular R π h ( e ) ≥ R π ( e ) R_{\pi}^{h}(e)\geq R_{\pi}(e) .
###### Assumption E.3 (Internalizability of hinted behavior) .
Hinted behavior is attainable without the hint: for every policy π ∈ Π \pi\in\Pi there exists a policy π ′ ∈ Π \pi^{\prime}\in\Pi with R π ′ ( e ) = R π h ( e ) for all e ∈ ℳ . R_{\pi^{\prime}}(e)=R_{\pi}^{h}(e)\qquad\text{for all }e\in\mathcal{M}.
### E.2 Main Results
Theorem E.5 shows that every pure Nash equilibrium in this setup drives hint-regret to zero, so that the Reasoning Agent is optimal on every environment in ℳ \mathcal{M} . The ideal hint-regret enables this result, because it vanishes on environments the Reasoning Agent already solves and on unsolvable environments, where the hint cannot help, and is positive only where the Reasoning Agent fails unaided but a hint would close the gap. The Environment Designer therefore profits exactly by targeting this learnable frontier, and at equilibrium no gap can remain anywhere, since the Environment Designer could otherwise deviate to it profitably. This is what distinguishes the regret reward from the plain adversarial-difficulty reward − 𝔼 e ∼ D [ R π ( e ) ] -\mathbb{E}_{e\sim D}[R_{\pi}(e)] , which provides no mechanism restricting the Environment Designer ’s support to useful environments.
###### Lemma E.4 (Hint-regret equals hint-free regret) .
Under Assumption E.2 , for any Reasoning Agent policy π ∈ Π \pi\in\Pi and any environment e ∈ ℳ e\in\mathcal{M} , R π h ( e ) − R π ( e ) = R ⋆ ( e ) − R π ( e ) . R_{\pi}^{h}(e)-R_{\pi}(e)=R^{\star}(e)-R_{\pi}(e). In particular the hint-regret is nonnegative, and strictly positive if and only if R π ( e ) < R ⋆ ( e ) R_{\pi}(e)<R^{\star}(e) .
###### Proof.
By Assumption E.2 , R π h ( e ) = R ⋆ ( e ) R_{\pi}^{h}(e)=R^{\star}(e) ; subtracting R π ( e ) R_{\pi}(e) gives the identity. The remaining claims follow since R ⋆ ( e ) ≥ R π ( e ) R^{\star}(e)\geq R_{\pi}(e) . ∎
###### Theorem E.5 (Nash equilibria imply hint-free optimality on every environment) .
Under Assumptions E.1 – E.3 , every pure Nash equilibrium ( D ∘ , π ∘ ) (D^{\circ},\pi^{\circ}) of the expected-regret self-play game satisfies u D ( D ∘ , π ∘ ) = 0 . u_{D}(D^{\circ},\pi^{\circ})=0. Moreover, the pointwise regret vanishes everywhere: for every e ∈ ℳ e\in\mathcal{M} , R π ∘ ( e ) = R ⋆ ( e ) , R π ∘ h ( e ) = R π ∘ ( e ) . R_{\pi^{\circ}}(e)=R^{\star}(e),\qquad R_{\pi^{\circ}}^{h}(e)=R_{\pi^{\circ}}(e).
Such equilibria exist: Assumptions E.2 and E.3 guarantee a uniformly optimal policy π ⋆ \pi^{\star} with R π ⋆ ( e ) = R ⋆ ( e ) R_{\pi^{\star}}(e)=R^{\star}(e) for all e e , and any pair ( D , π ⋆ ) (D,\pi^{\star}) is a pure Nash equilibrium, so the statement is not vacuous.
###### Proof.
Let ( D ∘ , π ∘ ) (D^{\circ},\pi^{\circ}) be a pure Nash equilibrium. By Lemma E.4 , ρ reg ( e ) := R π ∘ h ( e ) − R π ∘ ( e ) = R ⋆ ( e ) − R π ∘ ( e ) ≥ 0 for every e ∈ ℳ . \rho^{\text{reg}}(e):=R_{\pi^{\circ}}^{h}(e)-R_{\pi^{\circ}}(e)=R^{\star}(e)-R_{\pi^{\circ}}(e)\geq 0\qquad\text{for every }e\in\mathcal{M}.
Reasoning Agent side. By Assumption E.3 , there is π ′ ∈ Π \pi^{\prime}\in\Pi with R π ′ ( e ) = R π ∘ h ( e ) R_{\pi^{\prime}}(e)=R_{\pi^{\circ}}^{h}(e) for all e e ; by Assumption E.2 , R π ∘ h ( e ) = R ⋆ ( e ) R_{\pi^{\circ}}^{h}(e)=R^{\star}(e) , so R π ′ ( e ) = R ⋆ ( e ) ≥ R π ∘ ( e ) R_{\pi^{\prime}}(e)=R^{\star}(e)\geq R_{\pi^{\circ}}(e) for every e ∈ ℳ e\in\mathcal{M} . Hence u A ( D ∘ , π ′ ) = 𝔼 e ∼ D ∘ [ R ⋆ ( e ) ] ≥ 𝔼 e ∼ D ∘ [ R π ∘ ( e ) ] = u A ( D ∘ , π ∘ ) . u_{A}(D^{\circ},\pi^{\prime})=\mathbb{E}_{e\sim D^{\circ}}[R^{\star}(e)]\geq\mathbb{E}_{e\sim D^{\circ}}[R_{\pi^{\circ}}(e)]=u_{A}(D^{\circ},\pi^{\circ}). Since π ∘ \pi^{\circ} is a best response to D ∘ D^{\circ} , equality holds, so 𝔼 e ∼ D ∘ [ R ⋆ ( e ) − R π ∘ ( e ) ] = 0 \mathbb{E}_{e\sim D^{\circ}}[R^{\star}(e)-R_{\pi^{\circ}}(e)]=0 . The integrand is nonnegative, hence ρ reg ( e ) = 0 \rho^{\text{reg}}(e)=0 for D ∘ D^{\circ} -almost every e e , and therefore u D ( D ∘ , π ∘ ) = 𝔼 e ∼ D ∘ [ ρ reg ( e ) ] = 0 . u_{D}(D^{\circ},\pi^{\circ})=\mathbb{E}_{e\sim D^{\circ}}[\rho^{\text{reg}}(e)]=0.