SPADE: Self-Play in Adaptive Synthetic Executable Environments — part 6/8

Cleaned-text code points [70489, 84238). Bibliography omitted here; source.md includes it.

The most relevant line of recent work uses LLMs as their own source of training data in self-play, with varying degrees of reliance on external data. At one extreme, fully data-free methods bootstrap from minimal seeds: Absolute Zero Reasoner (AZR) ( Zhao et al., 2025 ) has a single model (shared parameters) propose and solve code-reasoning triplets (deduction, abduction, induction) with proposer reward 1 − r ¯ solve 1-\bar{r}_{\text{solve}} computed from solver Monte-Carlo success rates verified by a code executor, bootstrapping from a single identity function. R-Zero ( Huang et al., 2025 ) instead instantiates a Challenger and a Solver as two independently optimized copies of the same base LLM, training the Challenger via GRPO with a self-consistency uncertainty reward; the paper reports that this two-model design outperforms a single-model variant in ablations but plateaus after a few iterations. PopuLoRA ( Castanyer et al., 2026 ) extends this asymmetric paradigm to co-evolving populations of LoRA-adapter teachers and students with cross-evaluation between sub-populations. In a related but mechanistically distinct line, G-Zero ( Huang et al., 2026 ) performs DPO-based self-distillation over hint-induced preference pairs: a Proposer trained via GRPO emits a hint h h for query q q , the Generator π G \pi_{G} produces both an unassisted response a hard ∼ π G ( ⋅ ∣ q ) a_{\text{hard}}\sim\pi_{G}(\cdot\mid q) and a hint-conditional response a assisted ∼ π G ( ⋅ ∣ q , h ) a_{\text{assisted}}\sim\pi_{G}(\cdot\mid q,h) , and DPO is applied with a assisted a_{\text{assisted}} as chosen and a hard a_{\text{hard}} as rejected; the Proposer’s reward is Hint- δ \delta , the per-token mean log-probability shift induced on a hard a_{\text{hard}} when the hint is prepended to the Generator’s context. All supervision is derived from a single model’s own outputs under hint vs no-hint contexts, placing the method closer to the self-rewarding and self-distillation line ( Chen et al., 2024 ; Yuan et al., 2024 ; Singh et al., 2023 ) than to verifier-grounded agent-environment co-evolution. Tool-R0 ( Acikgoz et al., 2026 ) adapts the same Challenger-Solver paradigm to tool use, training the generator with a solve-rate-band reward plus a semantic alignment score and the solver with soft tool-call matching, similarly finding that separate parameters outperform a shared-weight setup. Self-Questioning Language Models ( Chen et al., 2025c ) have a single model propose and solve problems from only a topic prompt, with the proposer rewarded for majority-vote calibration (problems neither always solved nor always missed). Language Self-Play ( Kuba et al., 2025 ) proposes a data-free framework where question generation and answer improvement reinforce each other. PasoDoble ( Zhang et al., 2025d ) pairs an adversarial proposer and solver grounded in a pretraining knowledge base, sustaining improvement beyond R-Zero’s plateau. At the other extreme, methods that retain limited seed data or curated evaluation sets include SeRL ( Fang et al., 2025b ) , which uses a few-shot-prompted question generator with a difficulty-band filter and trains only the solver via Reinforce++ on majority-vote rewards from 500 seed instructions, and Sundaram et al. (2026) , who ground the teacher’s reward in measured student improvement on curated target problems instead of proxy statistics. Self-Challenging Language Model Agents ( Zhou et al., 2025 ) let agents propose harder task variants for themselves and then attempt to solve them. Autodata ( Kulikov et al., 2026 ) generalizes the self-instruct line by casting an LLM agent as a meta-optimizing data scientist that iteratively constructs training data; its Agentic Self-Instruct instantiation selects examples by the separation between a strong and a weak reference solver, targeting learnable examples over merely hard ones, a frontier-targeting principle analogous to SPADE ’s hint-based regret but operating over offline data creation with fixed reference solvers, without the online co-evolution of a gradient-trained generator. Vision-Zero ( Wang et al., 2025a ) brings gamified multi-agent self-play (Who Is the Spy?) to vision-language models with iterative self-play policy optimization. SPELL ( Yang et al., 2025 ) co-trains a three-role self-play loop (questioner, responder, verifier) on long documents, using a history-memory curriculum to generate progressively harder questions for evolving long-context capabilities. SPC ( Chen et al., 2025b ) trains a self-play critic via adversarial games to improve LLM reasoning evaluation. TextArena ( Guertler et al., 2025 ) provides an open-source collection of 57+ competitive text-based game environments with a Gym-compatible API for LLM evaluation and self-play training. Sarkar et al. (2025) train LLMs for Among Us-style social deduction with multi-agent RL. Self-Play with Execution Feedback ( Dong et al., 2024 ) uses code execution to verify instruction-following in a self-play loop. SPIRAL ( Liu et al., 2025a ) introduces self-play on zero-sum games as a multi-agent, multi-turn RL framework that incentivizes reasoning through game-theoretic competition. SPICE ( Liu et al., 2025b ) extends self-play to corpus environments via information asymmetry: a Challenger mines documents from a large corpus to generate diverse reasoning tasks with document-grounded answers, while a Reasoner solves them without document access; the corpus prevents hallucination amplification and information-symmetry collapse seen in ungrounded self-play. SWE-RL ( Wei et al., 2026 ) applies RL to open software evolution tasks, and SSR ( Wei et al., 2025 ) trains software engineering agents through self-play. Anchored Self-Play ( Choi et al., 2026 ) has a single model play a bug-generator and a fixer for code repair with a difficulty-band reward, adding reference-bug mixing and embedding-similarity shaping to keep self-generated bugs on the realistic distribution, preventing drift off it, and GASP ( Jana et al., 2026 ) guides asymmetric self-play for coding LLMs with grounding in real data. Chae et al. (2025) provide a systematic analysis of when LLM self-play succeeds and when it fails, analyzing the “invisible leash” where generation quality is bounded by base-model capability. Shafayat et al. (2025) investigate whether large reasoning models can self-train and find that naive self-play degrades performance without careful curriculum control. Liu et al. (2026b) analyze when self-synthetic self-play actually improves, showing that sustained evolution requires the synthetic pipeline to guarantee learnable information gain.

Two themes emerge across this body of work. First, all data-free self-play methods generate tasks (a problem statement paired with a sparse terminal reward) rather than full environments with state transitions. Second, the generator in every data-free case is either frozen or trained with heuristic proxy rewards (learnability, difficulty, variance) that risk reward hacking and distributional drift. By contrast, SPADE generates full MDP environments (state space, transition function, reward function, and verification code) as executable Python, and trains the Environment Designer via RL with hint-based regret that is grounded in measured Reasoning Agent return rather than proxy statistics.

### H.2 Unsupervised Environment Design and Open-Endedness

Curriculum learning ( Bengio et al., 2009 ) established the principle that ordering training examples from easy to hard accelerates learning. Quality-Diversity (QD) algorithms, exemplified by MAP-Elites ( Mouret and Clune, 2015 ) , illuminate the search space by maintaining an archive of high-performing solutions across a structured behavior space, providing the algorithmic substrate for many open-ended environment-generation systems. Paired Open-Ended Trailblazer (POET) ( Wang et al., 2019 ) pioneered the paradigm of co-evolving environments alongside agents, using evolutionary search to grow a population of parameterized BipedalWalker terrains together with the policies that solve them, with explicit transfer attempts moving high-performing agents across environments. Unsupervised Environment Design (UED) extends this idea by automatically generating training environments instead of selecting from a fixed pool. PAIRED ( Dennis et al., 2020 ) formalized UED via a minimax regret objective: an adversary generates environment parameters to maximize the gap between an antagonist’s and a protagonist’s returns, producing curricula of increasing complexity while avoiding unsolvable environments. Dennis et al. (2020) proved that at Nash equilibrium, the protagonist plays a minimax regret policy, connecting UED to decision theory. Prioritized Level Replay (PLR) ( Jiang et al., 2021b ) replaced the learned adversary with a replay buffer that prioritizes high-regret levels, achieving comparable curriculum quality with lower computational cost. Replay-Guided Adversarial Environment Design ( Jiang et al., 2021a ) combined the adversarial generator with prioritized replay, using the replay distribution to guide the adversary toward high-regret regions of the environment space. ACCEL ( Parker-Holder et al., 2022 ) introduced evolutionary mutations of high-regret levels, compounding complexity over training without domain-specific heuristics and recovering minimax regret guarantees at a fraction of the compute required by population-based methods like POET. Mediratta et al. (2023) stabilized PAIRED-style learned adversaries via entropy regularization and behavioral cloning between protagonist and antagonist, addressing the entropy collapse that destabilizes UED. Rutherford et al. (2024) investigated regret approximations used in PLR and ACCEL, showing that common proxies (positive value loss, maximum Monte Carlo) correlate with success rate rather than true regret, and proposing Sampling For Learnability (SFL) as an improved estimator. Monette et al. (2025) recast UED as a nonconvex-concave optimization over a categorical level distribution, proving convergence to first-order Nash equilibria and generalizing learnability scores to continuous-return settings. CENIE ( Teoh et al., 2024 ) augmented regret-based UED with a novelty objective that measures how much a candidate environment pushes the agent into unexplored regions of the state-action space, finding that novelty and regret are synergistic rather than competing. DISCOVER ( Diaz-Bone et al., 2026 ) provides a complementary formal-analysis view from goal-conditioned RL: it scores candidate goals via current-policy value estimates balancing achievability, relevance, and novelty, and proves a UCB-style bound on time-to-target-achievability that depends only on the agent’s initial distance to the target, independent of task-space volume. The three-axis selection rule parallels SPADE ’s hint-based regret as a frontier-targeting curriculum signal, but DISCOVER operates over a fixed goal pool whereas SPADE generates the environment distribution itself.

A broader thread of open-endedness research motivates the need for unbounded environment spaces. Hughes et al. (2024) argue that open-endedness, defined as simultaneous novelty and learnability, is essential for superhuman AI, and that systems plateau when the environment parameterization is exhausted. Baker et al. (2019) demonstrated emergent tool use from multi-agent autocurricula in hide-and-seek, showing that competitive self-play in rich physics environments produces increasingly sophisticated strategies. OMNI ( Zhang et al., 2023 ) uses foundation models as a “Model of Interestingness” to filter open-ended task proposals, focusing the curriculum on tasks that are both learnable and interesting. OMNI-EPIC ( Faldor et al., 2024 ) extends this line by representing each environment as executable code generated by a foundation model, with a frozen FM judge of interestingness gating which generated tasks enter the archive; SPADE shares the code-as-environment representation but trains the Environment Designer via RL with hint-based regret, with no frozen interestingness judge. Imagined Autocurricula ( Güzel et al., 2025 ) applies PLR inside a learned diffusion world model trained from offline data, demonstrating that UED principles transfer to imagined environments. SIMA 2 ( Bolton et al., 2025 ) scales open-ended self-improvement to embodied 3D worlds: a Gemini-based agent uses foundation models to generate its own tasks and rewards and acquires new skills in previously unseen and even model-generated (Genie 3) environments, though it distributes the process across separate task-setter, agent, and reward models plus a distinct generative world model instead of a single policy that authors its own environments. Goldfeder et al. (2026) argue for Superhuman Adaptable Intelligence over AGI, emphasizing speed of adaptation over static benchmark performance. PAPRIKA ( Tajwar et al., 2025 ) trains generally curious agents on ten hand-designed task groups, showing that diverse multi-turn training produces transferable exploration strategies.

All classical UED methods operate in parameterized environment spaces (maze dimensions, terrain friction, grid layouts) with small, fixed design vocabularies. To our knowledge, SPADE is the first system to bring regret-based UED with an RL-trained Environment Designer that produces full MDP environments to LLM post-training: the environment space is unbounded (arbitrary Python code), the Environment Designer is the LLM rather than a separate adversary network, and hint-based regret provides the co-evolution signal without requiring a trained antagonist.

### H.3 Synthetic Environment Generation