SPADE: Self-Play in Adaptive Synthetic Executable Environments — part 1/8

Cleaned-text code points [0, 14309). Bibliography omitted here; source.md includes it.

1]University of Washington 2]Stanford University 3]Northeastern University 4]Carnegie Mellon University 5]Massachusetts Institute of Technology 6]National University of Singapore 7]Seoul National University 8]Stevens Institute of Technology 9]University of Chicago \contribution [*]Equal contribution \contribution [†]Joint last author \correspondence , , \metadata [Code] https://github.com/spade-rl/spade \metadata [Project] https://spade-rl.github.io

# SPADE ♠ \spadesuit : Self-Play in Adaptive Synthetic Executable Environments

###### Abstract

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset() / step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent ’s regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent’s capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by + 5.3 +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by + 5.7 +5.7 on BFCL v4 multi-turn and + 13.9 +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.

## 1 Introduction

Agentic AI has become broadly capable: language models now reason over long horizons, use tools, search the web, and operate computers ( OpenAI, 2024 ; DeepSeek-AI, 2025 ; Wang et al., 2025c ; Zhang et al., 2025c ) . These gains increasingly come not from pretraining alone but from learning through experience, where a model improves from its own trajectories of interacting with an environment and the rewards they return ( Silver and Sutton, 2025 ) . As high-quality human text is a finite resource ( Villalobos et al., 2024 ) , the bottleneck for further agentic progress is increasingly the supply of training environments: interactive tasks with verifiable rewards. Building them has become a central industry investment, with major labs weighing environment budgets exceeding $1 billion a year and a new class of startups raising nine figures to supply them ( Zeff, 2025 ; PYMNTS, 2026 ) . Yet whether hand-built or synthesized, these environments form a fixed pool that does not adapt as the learner improves, so an agent stops improving once it exhausts them.

Several lines of work try to keep agents improving, each with its own limitation. Harness engineering improves how an agent acts within a given environment at inference time, through tool orchestration and self-correction ( Yao et al., 2023 ; Shinn et al., 2023 ) , but it changes no weights, so its gains are tied to each hand-built harness, rather than improving general capabilities the model carries to new tasks. Human-curated scaling ( Zhang et al., 2025a ; Guertler et al., 2025 ; Liu et al., 2025c ) builds diverse environments to train across, but scales only as fast as people can write them. Synthetic generation ( Wang et al., 2026b ; Tu et al., 2026 ; Zhu et al., 2026 ; Gandhi et al., 2026 ; Zeng et al., 2025 ) produces environments programmatically, but the generators are fixed, so the environment space does not grow with the agent and the model soon exhausts it ( Song et al., 2024 ) . Self-play ( Zhao et al., 2025 ; Huang et al., 2025 ; Liu et al., 2025a ) lets a model improve through dual roles, but ungrounded self-play is bounded by information symmetry, unable to pose challenges beyond its own knowledge and prone to amplifying its errors ( Chae et al., 2025 ) ; corpus grounding mitigates this ( Liu et al., 2025b ) , yet most methods still generate tasks (a problem with a sparse terminal reward) rather than complete multi-turn environments (state transitions, reward functions, and verification code). None produces a self-improving system whose environments keep growing in complexity as the agent improves.

We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a framework where a single LLM plays two roles: an Environment Designer that produces complete training environments as executable Python code, and a Reasoning Agent that learns from them. Each environment implements a Gym-style interface (the standard reset() / step() API used in reinforcement learning) ( Brockman et al., 2016 ) , representing a full Markov decision process (MDP) with state transitions and reward functions. This code-as-environment representation unifies single-turn settings (one step to terminal reward) and multi-turn agentic tasks (sequential interaction with transition dynamics) under a single interface. Because any computable MDP can be written as a program, the Environment Designer can express any such environment in code, rather than only those a hand-designed parameterization allows. Unlike prior works where the environment generator is frozen, SPADE ’s Environment Designer is itself trained via RL with a hint-based regret signal that targets solvable environments precisely at the frontier of the Reasoning Agent ’s capability, creating co-evolution where the environment distribution shifts as the Reasoning Agent improves (Figure 2 ).

Overall, our work makes the following contributions: 1. We introduce a general framework for co-evolving environments synthesis and agentic capability through self-play , where the same LLM both generates executable environments as code and learns to solve them. By representing environments as Python programs with a Gym-style interface, the framework unifies single-turn reasoning and multi-turn agentic tasks, and turns environment design into a learnable, RL-trained component of post-training, enabling continual open-ended self-improvement.

2. We introduce a hint-based regret reward for the Environment Designer . The Environment Designer is rewarded by the gap between Reasoning Agent performance with and without privileged hints, a lightweight estimate of minimax regret ( Dennis et al., 2020 ) that targets environments at the frontier of the Reasoning Agent ’s capability. Unlike a pure adversary, which is free to make environments unsolvable, or a cooperative designer, which can inflate Reasoning Agent reward without teaching it anything, hint-based regret yields constrained competitive dynamics that reward environments which are both solvable and at the learning frontier, with an equilibrium analysis in Appendix E .

3. We design a corpus-grounded and memory-augmented design pipeline spanning cognitive-skill games and tool-use tasks. Where prior environment-generation work targets a single domain (math, code, or tool use), SPADE demonstrates improvements in both settings, with environments grounded in pretraining corpus knowledge and accumulated experience.

4. We validate self-play working at 30B+ scale with practical recipe provided, demonstrating that the approach works beyond the small-model with a complete training recipe including environment validation, reward hacking avoidance, and curriculum design.

On the games setting, SPADE outperforms fixed-environment baselines on all three backbones, by + 5.3 +5.3 points on average and up to + 7.5 +7.5 on individual benchmarks over the strongest fixed-environment baseline at 30B-A3B. In the tool-use setting, SPADE lifts BFCL v4 multi-turn ( Patil et al., 2025 ) by + 10.3 +10.3 at 4B and + 5.7 +5.7 at 30B-A3B, and ACEBench-Agent ( Chen et al., 2025a ) by + 13.9 +13.9 at 30B-A3B. Ablations show the advantage of hint-based regret over an exponential-moving-average (EMA) learning-potential signal ( Kanitscheider et al., 2021 ; Zhang et al., 2023 ) , and of the full adaptive, corpus-grounded, memory-augmented configuration over the partial and non-adaptive controls. Qualitative analysis reveals emergent curricula: SPADE progresses from simple single-skill tasks to complex multi-constraint environments requiring long-horizon interaction.

In SPADE , the Environment Designer writes a new, self-contained environment and the Reasoning Agent learns by acting in it. Making design a learnable role lets the two co-evolve: as the agent improves, so do the environments it trains on (Figure 2 ). A single model thus improves not only how it reasons and acts, but the worlds it builds to learn in, a concrete step toward open-ended, continual self-improvement.

## 2 Related Work

For an extended background and comparison to the literature see Appendix H , summarized here:

Self-Play for LLMs Self-play has driven capability growth from TD-Gammon ( Tesauro et al., 1995 ) through AlphaZero ( Silver et al., 2017 ; Silver et al., 2018 ) to modern game AI ( Vinyals et al., 2019 ; † et al.(2022)(FAIR)†, Bakhtin, Brown, Dinan, Farina, Flaherty, Fried, Goff, Gray, Hu, et al., FAIR ) . PowerPlay ( Schmidhuber, 2013 ) proposed a single self-modifying system that continually invents tasks at its own capability frontier, prefiguring the dual-role single-LLM pattern adopted below. Asymmetric self-play ( Sukhbaatar et al., 2017 ) later formalized a proposer-solver paradigm in which one agent sets challenges while another solves them. Applying self-play to LLMs is harder than in single-task game-playing systems: the goal is general capability across open-ended problems, and self-play training at LLM scale is often unstable. Earlier LLM self-play methods rely on curated seed data or evaluation sets, either for self-distillation ( Chen et al., 2024 ; Yuan et al., 2024 ; Singh et al., 2023 ) , for adversarial language games over fixed vocabularies ( Cheng et al., 2024 ) , or for bootstrapping a learned proposer ( Fang et al., 2025b ; Sundaram et al., 2026 ) . A more recent line removes this dependency, training proposer and solver from minimal seeds with proxy rewards that target the learnable frontier. AZR ( Zhao et al., 2025 ) , SQLM ( Chen et al., 2025c ) , and Language Self-Play ( Kuba et al., 2025 ) have a single LLM play both roles with shared parameters; R-Zero ( Huang et al., 2025 ) and Tool-R0 ( Acikgoz et al., 2026 ) instead train two separate models initialized from the same base, using uncertainty- or solve-rate-based rewards. The closest predecessors to our work are SPIRAL ( Liu et al., 2025a ) , which trains LLMs through self-play on multi-turn zero-sum language games, and SPICE ( Liu et al., 2025b ) , which extends self-play to corpus-grounded question generation. However, both generate tasks (a single problem with only a terminal reward) and can collapse when the model’s capacity bounds the reachable solution distribution ( Chae et al., 2025 ) or when proxy rewards induce degenerate outputs ( Shafayat et al., 2025 ) ; a recent theorem-proving method adds a frozen Guide to score the proposer’s conjectures and curb collapse (SGS, Bailey et al., 2026 ); in SPADE the verifier is instead part of each generated environment, co-evolving with the Environment Designer rather than fixed. By contrast, SPADE generates full multi-turn MDP environments (state transitions, reward functions, and verification code) as executable code, trains the Environment Designer itself via RL with hint-based regret grounded in measured Reasoning Agent return, and spans a broader range of tasks (games and tool use) than prior methods confined to a single domain.

Unsupervised Environment Design and Open-Endedness Curriculum learning and learnability-driven adaptation ( Bengio et al., 2009 ; Schmidhuber, 2013 ) laid the groundwork for what Dennis et al. (2020) termed unsupervised environment design (UED). POET ( Wang et al., 2019 ) pioneered paired open-ended environment-agent co-evolution via evolutionary search over parameterized terrain, while PAIRED ( Dennis et al., 2020 ) introduced adversarial environment design with minimax regret; subsequent work improved level curation via prioritized replay ( Jiang et al., 2021b ; Jiang et al., 2021a ) , evolutionary complexity ( Parker-Holder et al., 2022 ) , stability ( Mediratta et al., 2023 ) , regret approximations ( Rutherford et al., 2024 ) , and theoretical foundations ( Monette et al., 2025 ) . Güzel et al. (2025) extends prioritized level replay to imagined trajectories from a learned diffusion world model. Multi-agent autocurricula provide a complementary route to emergent complexity ( Baker et al., 2019 ) , and open-endedness has been argued essential for superhuman AI ( Hughes et al., 2024 ) . OMNI-EPIC ( Faldor et al., 2024 ) represents environments as executable code generated by foundation models, with a frozen FM judge of interestingness gating task acceptance. PAPRIKA ( Tajwar et al., 2025 ) brings curriculum-style training to LLM agents across diverse synthetic task groups. However, classical UED searches parameterized environments (maze dimensions, grid layouts) within small, fixed design spaces, and curriculum-based LLM training draws from fixed task pools without a learned generator. SPADE carries the regret principle into an unbounded, code-defined environment space: the Environment Designer is itself the learning agent, writing full MDPs as programs and co-evolving with the Reasoning Agent online, rather than an adversary selecting from a hand-designed parameterization. This turns environment design from selection within a fixed space into open-ended generation.