SPADE: Self-Play in Adaptive Synthetic Executable Environments — part 5/8

Cleaned-text code points [60799, 70489). Bibliography omitted here; source.md includes it.

Environment Designer side. Fix any e ∈ ℳ e\in\mathcal{M} . The Environment Designer may deviate to the point mass δ e ∈ Δ ⁡ ( ℳ ) \delta_{e}\in\Delta(\mathcal{M}) , which yields u D ​ ( δ e , π ∘ ) = ρ reg ​ ( e ) u_{D}(\delta_{e},\pi^{\circ})=\rho^{\text{reg}}(e) . Since ( D ∘ , π ∘ ) (D^{\circ},\pi^{\circ}) is a Nash equilibrium, ρ reg ​ ( e ) = u D ​ ( δ e , π ∘ ) ≤ u D ​ ( D ∘ , π ∘ ) = 0 . \rho^{\text{reg}}(e)=u_{D}(\delta_{e},\pi^{\circ})\leq u_{D}(D^{\circ},\pi^{\circ})=0. Combined with ρ reg ​ ( e ) ≥ 0 \rho^{\text{reg}}(e)\geq 0 from Lemma E.4 , this gives ρ reg ​ ( e ) = 0 \rho^{\text{reg}}(e)=0 for every e ∈ ℳ e\in\mathcal{M} . Therefore R π ∘ ​ ( e ) = R ⋆ ​ ( e ) , R π ∘ h ​ ( e ) = R π ∘ ​ ( e ) R_{\pi^{\circ}}(e)=R^{\star}(e),\qquad R_{\pi^{\circ}}^{h}(e)=R_{\pi^{\circ}}(e) for every e ∈ ℳ e\in\mathcal{M} . ∎

## Appendix F Method and Experiment Details [back to contents]

### F.1 System Prompts and Templates

##### Environment-generation prompt.

The Environment Designer receives the prompt below to generate a new single-turn game as executable Python; the skill name, its description, and a short list of example concepts fill the slots shown in angle brackets. In the canonical corpus-grounded runs the Environment Designer instead conditions on a freshly sampled corpus document, and the example-concept slot is unused. The single-turn prompt serves the single-turn setting of Section 3.1 (one derivation task, graded per attempt), while the multi-turn prompts demand stateful interaction. The \boxed{} extraction in the displayed contracts stops at the first closing brace, so generated environments use plain, brace-free answer strings.

##### Multi-turn environment-generation prompt.

Because gameplay spans multiple turns, the Environment Designer also receives a dedicated prompt for generating interactive, multi-turn games (state that evolves across turns, branching actions), shown below with the same placeholder convention.

##### Corpus-grounded interactive generation prompt.

The canonical games runs use the corpus-grounded variant below: the sampled document replaces the example-concept list, and the specification demands hidden state, multi-turn structure, and partial reward. The skill, difficulty, and document slots are shown in angle brackets.

##### Tool-use generation prompt.

The tool-use runs generate environments with the multi-turn variant below (selected by SPARE_MULTITURN_ENV_GEN=1 in the released launchers). The skill, difficulty, and example slots fill as in the games prompts; each generation call additionally samples one of ten application domains and one of four task variants (weighted 3 : 2 : 1 : 1 3{:}2{:}1{:}1 ), whose filler blocks follow the main template.

##### Reasoning Agent gameplay prompt.

The Reasoning Agent receives each turn’s observation wrapped in the template below and answers in \boxed{} form.

### F.2 Implementation Details

Two auxiliary Environment Designer reward terms are disabled in all full- SPADE runs: a per-environment EMA-based learning-potential bonus ( Kanitscheider et al., 2021 ; Zhang et al., 2023 ) , enabled only as the standalone reward replacement evaluated in Section 7.2 , and a frontier bonus (extra reward for environments with a large fast-versus-slow EMA gap | μ fast − μ slow | |\mu_{\text{fast}}-\mu_{\text{slow}}| ). No separate variance bonus (a r ¯ A ​ ( 1 − r ¯ A ) \bar{r}_{A}(1-\bar{r}_{A}) -style reward for mixed outcomes) is used. Failed environment candidates are regenerated up to a fixed number of attempts. Table 5 lists the full training configuration.

Pool lifecycle. At each regeneration the previous environment set is deleted and replaced; the environment memory evicts oldest-first at its 200 200 -record cap; rejected candidates are persisted for inspection; generation retries up to five attempts per environment.

Hint generation detail. The privileged hint is produced by a separate designer-side call conditioned on the generated environment’s source code: the hint writer sees the code, while the Reasoning Agent never does.

Validation detail. Every candidate environment must pass a programmatic smoke test: the class is instantiated, reset() is called, and a few probe actions are stepped through, discarding candidates that fail to parse or crash. The tool-use setting adds the two semantic checks of Section 5.3 : a deterministic reset gate that rejects environments in which a success criterion errors on the freshly reset state under every tested seed, and an LLM screen that rejects impossible environments (unreachable, pre-satisfied, or underivable success criteria) while keeping hard-but-feasible ones.

### F.3 Reproducibility

The training and evaluation code is released. The configuration files for every run reported here are released with the paper. The evaluation output JSONs and the scripts that produce every figure are included alongside the code.

#### F.3.1 Comparability Notes for Table 2

The reference systems in Table 2 were trained and evaluated by their own authors on their own harnesses; we reprint their published scores for context and record the protocol differences here. BFCL versions. Our rows and Agent-World ( Dong et al., 2026 ) report the BFCL v4 multi-turn suite. AWM ( Wang et al., 2026b ) and EnvScaler ( Song et al., 2026 ) label the suite v3, which contains the same four multi-turn subcategories; EnvScaler evaluates a frozen 2024-09-22 data snapshot whose repository notes minor differences from later releases. τ 2 \tau^{2} -bench variants. AWM evaluates τ 2 \tau^{2} -bench-verified, a fork with task corrections, runs the Telecom domain in solo mode (agent only, no user simulator), and uses a GPT-5.1 user simulator in the other domains; AgentScaler ( Fang et al., 2025a ) does not state its user simulator. We evaluate the standard τ 2 \tau^{2} -bench release. Aggregation. Reference-row Avg cells follow the rule stated in the table caption; Agent-World and AWM print their own aggregates (61.8/65.4 and task-weighted 33.5/39.0), which do not equal the unweighted means of the domain scores they report. Base models. Our rows post-train Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507; the reference systems post-train their own base checkpoints, with training data and budgets that differ from ours.

## Appendix G Extended Ablations [back to contents]

Cross-family transfer. As a check on a non-Qwen backbone, we ran the same recipe on Nemotron-30B-A3B-BF16, evaluating only the Reasoning-Gym suite; AIME 2025/2026, GPQA-Diamond, and LiveCodeBench-v6 were not evaluated on this backbone, so this checks that the training dynamics are not Qwen-specific rather than serving as a matched cross-family comparison. All four Reasoning-Gym categories rise above the untrained base, with the largest Nemotron gains in RG-Cognition ( + 9.6 +9.6 ) and RG-Algorithmic ( + 9.3 +9.3 ; Figure 16 ).

Table 6 expands the component-controlled ablations of Table 3 to every games-setting variant reported in this paper, including the skill-diversity run, with each variant’s selected checkpoint and its GEM overall win rate. Checkpoints are selected per variant by the best suite average (the mean of the eight benchmark columns); the corresponding evaluation trajectories appear in Figures 11 , 15 , and 12 .

## Appendix H Extended Related Work [back to contents]

### H.1 Self-Play for LLMs

Self-play has been a cornerstone of AI since Tesauro et al. (1995) trained TD-Gammon through self-play in backgammon. AlphaGo ( Silver et al., 2016 ) combined Monte Carlo tree search with deep neural networks and self-play to defeat a human Go champion, and AlphaZero ( Silver et al., 2017 ; Silver et al., 2018 ) extended the approach to chess, shogi, and Go from tabula rasa with no human data. OpenAI Five ( Berner et al., 2019 ) and AlphaStar ( Vinyals et al., 2019 ) scaled multi-agent self-play to Dota 2 and StarCraft II, while Cicero ( † et al.(2022)(FAIR)†, Bakhtin, Brown, Dinan, Farina, Flaherty, Fried, Goff, Gray, Hu, et al., FAIR ) combined language models with strategic reasoning in Diplomacy. Littman (1994) formalized the Markov-game framework underlying these results, and Irving et al. (2018) proposed debate as an AI-safety mechanism grounded in self-play. The asymmetric self-play paradigm of Sukhbaatar et al. (2017) , in which one agent proposes challenges while another solves them, provides the conceptual template for teacher-student environment design.

Translating self-play to LLMs introduces new challenges because rewards are sparse, outputs are discrete sequences, and the “environment” is open-ended language. SPIN ( Chen et al., 2024 ) applies self-play fine-tuning by training the LLM to distinguish its own outputs from human demonstrations, converting weak models to strong ones. Self-Rewarding Language Models ( Yuan et al., 2024 ) let the LLM judge its own outputs to generate preference data, iteratively improving both generation and evaluation. SPAG ( Cheng et al., 2024 ) uses adversarial language games (Adversarial Taboo) to incentivize strategic reasoning through a zero-sum objective. ReST EM ( Singh et al., 2023 ) alternates between sampling model solutions filtered by a binary correctness reward and supervised fine-tuning on the filtered correct rollouts. Prover-Verifier Games ( Kirchner et al., 2024 ) train a prover to produce legible solutions that a weaker verifier can check, improving output interpretability. ReMA ( Wan et al., 2025 ) decomposes reasoning via multi-agent RL into a high-level meta-thinking agent (strategy and decomposition) and a low-level reasoning agent (step-by-step execution), trained jointly.