SPADE: Self-Play in Adaptive Synthetic Executable Environments — part 7/8

Cleaned-text code points [84238, 99159). Bibliography omitted here; source.md includes it.

A growing body of work uses LLMs to programmatically generate training environments for agentic RL, addressing the bottleneck of hand-curated environment design. Agent World Model (AWM) ( Wang et al., 2026b ) decomposes environment synthesis into five stages (scenario, task, database, interface, verification) to produce 1,000 SQLite-backed tool-use environments with MCP interfaces, training agents via GRPO with hybrid step-level and task-level rewards. ScaleEnv ( Tu et al., 2026 ) generates multi-turn tool-use environments by synthesizing API specifications, databases, and verification code from seed domains, scaling to thousands of environments for generalist interactive agent training. TermiGen ( Zhu et al., 2026 ) synthesizes high-fidelity terminal environments inside Docker containers with automated trajectory verification, producing diverse command-line tasks for terminal agent training. Nemotron-Terminal ( Pi et al., 2026 ) introduces Terminal-Task-Gen, a two-stage pipeline that combines dataset adaptation (transforming math, code, and SWE benchmarks into terminal-formatted prompts) with synthetic task generation (seed-based and skill-based, the latter driven by a Skill Taxonomy of nine domains and primitive skills); the resulting Terminal-Corpus is open-sourced, and Qwen3-32B post-trained via SFT reaches 27.4% on Terminal-Bench 2.0 (from a 3.4% baseline), surpassing Qwen3-Coder-480B (23.9%) at a fraction of the parameter count. SkillSynth ( Fan et al., 2026 ) constructs a scenario-mediated skill graph with 82,073 scenarios as nodes, 57,214 filtered skills as directed transitions, and 185,529 LLM-verified bridges, samples directed paths through the graph as workflow abstractions, and instantiates them via a multi-agent harness with dual execution-based and rubric-based verification (95.7% oracle pass rate, 3,560 verified task instances per run); the explicit objective is maximizing the diversity of execution trajectories over raw task count. Eurekaverse ( Liang et al., 2024 ) uses LLMs to generate Python terrain-program curricula (height-field functions) for robot skill learning, producing progressively harder physics environments guided by agent performance feedback, and validates on quadrupedal parkour. LLM-in-Sandbox ( Cheng et al., 2026 ) places LLMs in sandboxed code-execution environments and shows that agentic intelligence emerges from multi-turn interaction with executable feedback. EvoCUA ( Xue et al., 2026a ) evolves computer-use agents by synthesizing scalable GUI interaction experiences, generating training trajectories across diverse desktop applications. Xue et al. (2026b) extend this to autonomous continual learning where computer-use agents adapt to new environments through self-generated experience. DreamGym ( Chen et al., 2025e ) trains an LLM-based world model on demonstration trajectories from existing agentic environments (WebShop, ALFWorld, WebArena) and uses it to generate abstract-text rollouts that replace expensive real-environment interactions during RL. Simia ( Li et al., 2025 ) pushes this direction further by removing real environments entirely: Simia-SFT amplifies small seed trajectories into diverse SFT data via reasoning-model-simulated feedback, and Simia-RL performs PPO/GRPO directly against LLM-generated environment transitions; fine-tuned 7–32B Qwen and Llama models reach 36–59 average on τ 2 \tau^{2} -bench (Airline+Retail subset) without ever executing real tool code, but the framework still inherits the LLM-as-transition-model hallucination risk and the reduced two-domain τ 2 \tau^{2} setup is not directly comparable to the three-domain Avg used elsewhere. Lu et al. (2025) argue that tuning the environment (reward shaping, observation design) can be more effective than tuning the agent, providing a complementary perspective on environment optimization. Exploratory Iteration ( Jiang et al., 2025b ) grows a self-improvement task space by sampling informative intermediate solution iterates from previous episodes as starting points for new single-step training tasks, training K K -step inference-time self-improvement while only training on single-step transitions. Golden Goose ( Lu et al., 2026 ) synthesizes RLVR tasks from unverifiable internet text by extracting verifiable sub-claims, converting passive corpora into training signal without requiring curated datasets. AutoEnv ( Zhang et al., 2025b ) provides automated environments for measuring cross-environment agent transfer, enabling systematic evaluation of generalization. Synthetic Sandbox ( Zhou et al., 2026 ) generates sandboxed ML-engineering environments for training software agents on realistic development workflows. RLAnything ( Wang et al., 2026a ) proposes a dynamic RL system that jointly optimizes policy and reward model while adapting the task distribution via perturbation of existing tasks. GenEnv ( Guo et al., 2025 ) targets difficulty-aligned environment generation with a curriculum reward signal. Agent-World ( Dong et al., 2026 ) mines real-world environments and toolsets from web content (1,978 environments, 19,822 tools) and applies a self-evolving training arena that diagnoses agent weaknesses and generates targeted tasks across rounds. AgentScaler ( Fang et al., 2025a ) clusters over 30,000 real APIs into more than 1,000 simulated tool-use domains via Louvain community detection on a parameter-similarity graph, materializes each tool as a Python class operating on a per-domain database schema, and trains 4B/8B/30B-A3B agents via two-phase supervised fine-tuning on filtered agent-user trajectories without RL. EnvScaler ( Song et al., 2026 ) programmatically synthesizes tool-interactive environments for multi-turn tool-use agent training, and EnvFactory ( Xu et al., 2026b ) scales executable tool-use environment synthesis paired with a stable RL recipe. From Trainee to Trainer ( Chen et al., 2026 ) narrows the adaptivity gap by letting the current RL checkpoint reconfigure its own environment generator from diagnosed failures across training rounds, though it tunes a fixed set of generator parameters and does not emit executable environment code. AutoPlay ( Ramrakhya et al., 2025 ) addresses task synthesis for UI agents (mobile and desktop) by first running an MLLM explorer in each app to gather state and functionality information, then having a frozen GPT-4o task-generator condition on those exploration trajectories plus task-guideline prompts to propose grounded tasks; the executor is trained via SFT and GRPO with binary outcome rewards from an MLLM verifier, gaining 20.6 points on AndroidWorld over Qwen2.5-VL-7B. InfiniteWeb ( Zhang et al., 2026e ) synthesizes complete static-HTML/localStorage websites alongside tasks and dense-reward JavaScript evaluators via task-centric test-driven development; UI-TARS-1.5-7B post-trained with GRPO on 600 generated tasks improves on OSWorld from 24.5 to 31.4 and on Online-Mind2Web from 23.0 to 28.7, with the largest gains concentrated in easy and medium difficulty buckets. EvoEnv ( Shi et al., 2026 ) co-trains a single LLM under generator and solver role-conditioning via shared-parameter GRPO, producing reusable Python verifier-and-prompt artifacts whose oracle and scorer remain frozen at training time; the generator reward combines staged validation, solver-relative difficulty calibration targeting 30% solver accuracy, and an embedding-based novelty bonus, with evaluation on three model families restricted to single-shot reasoning benchmarks (Qwen3-4B-Thinking-2507 improves from 72.4 to 74.8 averaged across eight math, code, and science benchmarks). The environment interface is single-shot scoring (sampler, oracle, renderer, scorer) without state evolution or step-level rewards, so the framework does not extend to multi-turn agentic settings such as tool use or terminal workflows; SPADE differs in three respects: (i) the Environment Designer is rewarded via hint-based regret grounded in Reasoning Agent return rather than validation and novelty signals, (ii) environments are full MDPs with reset/step interfaces that unify single-turn and multi-turn settings, and (iii) evaluation spans games and tool use in addition to reasoning.

These methods produce useful training material, and some incorporate feedback-driven adaptation where the generation distribution shifts in response to agent progress: Eurekaverse evolves environments based on agent performance statistics, and EvoCUA iterates synthesis with an improving policy. However, even in these cases the generator LLM receives no RL gradients; adaptation operates through prompting heuristics or iterative data filtering, without joint optimization. By contrast, SPADE ’s Environment Designer is a genuine RL learner, trained via hint-based regret in the same optimization loop as the Reasoning Agent , producing full MDP environments at the Reasoning Agent ’s competence frontier.

### H.4 Environment Scaling

A growing consensus holds that the next frontier for agentic RL is scaling environments rather than algorithms. Silver and Sutton (2025) articulate this vision most directly, arguing that the field is entering an “era of experience” in which the primary bottleneck is generating sufficiently diverse and abundant training environments. Zhang et al. (2026b) argue in the same vein that scalable, diverse environments are what drive generalizable agents. AgentRL ( Zhang et al., 2025a ) provides a multi-turn, multi-task RL framework with a fully asynchronous generation-training pipeline, cross-policy sampling for improved exploration, and task advantage normalization, training a single agent across five agentic domains to outperform frontier models. SCALER ( Xu et al., 2026a ) converts competitive programming problems (from CodeContests) into parameterized single-turn reasoning environments at controlled difficulty levels, demonstrating that scaling the number and diversity of training problems yields consistent improvements in mathematical reasoning. RLVE ( Zeng et al., 2025 ) creates 400 hand-engineered verifiable environments for RL training with adaptive difficulty levels and per-environment custom verifiers (mixing rule-based parsing, soft scoring, and code execution) in place of LLM-as-judge. WebScale-RL ( Cen et al., 2025 ) builds an automated data pipeline that harvests web content to create RL training environments at pretraining scale, showing that more diverse environments produce better generalization. Endless Terminals ( Gandhi et al., 2026 ) procedurally generates 3,255 verified terminal-use tasks via a four-stage pipeline and demonstrates that vanilla PPO with binary rewards yields substantial gains as the number of training environments scales, with transfer to TerminalBench 2.0. Self-Evolving Curriculum ( Chen et al., 2025d ) dynamically adjusts the difficulty distribution of training problems based on the learner’s current performance, preventing saturation on easy examples. TTCS ( Yang et al., 2026 ) co-evolves a gradient-trained problem synthesizer with a solver via GRPO, using a capability-adaptive reward that targets the solver’s learning frontier at test time. Xu et al. (2025) survey the path toward Large Reasoning Models, organizing the field into three pillars (automated data construction, learning-to-reason via RL, and test-time scaling) and identifying environment scaling as a key open problem. Song et al. (2024) examine self-improvement capabilities of LLMs and find that naive self-training saturates quickly, motivating the need for adaptive curricula. Embodied Co-Design ( Wang et al., 2025b ) provides a taxonomy of co-design approaches where agent morphology and controller evolve jointly, drawing parallels to SPADE ’s co-evolution of environment and agent. Agent Learning via Early Experience ( Zhang et al., 2025c ) introduces a reward-free training paradigm in which agents propose alternative actions at expert-visited states and learn from the resulting future states (implicit world modeling and self-reflection), bridging imitation learning and full RL across eight agentic environments.

These works independently demonstrate that more environments yield better generalization and that curriculum design matters at least as much as algorithm choice. Several incorporate adaptive mechanisms: RLVE adjusts per-environment difficulty levels, SCALER tracks accuracy for difficulty control, and Self-Evolving Curriculum learns a sampling policy over problem categories. SPADE combines gradient-trained environment design with the code-as-environment representation, producing an adaptive curriculum of executable MDPs that co-evolves with the Reasoning Agent .

### H.5 Agentic Memory Design and Self-Improving Code Systems

MemRL ( Zhang et al., 2026d ) introduces self-evolving agents that learn via runtime reinforcement learning on episodic memory, storing and retrieving past interaction experiences to improve future decision-making without additional gradient updates. ALMA ( Xiong et al., 2026 ) meta-learns agentic memory designs themselves: a foundation model searches the space of Python memory-architecture programs (an open-ended scaffold over update/retrieve operations), discovering memory designs that improve continual learning at test time. HyperAgents ( Zhang et al., 2026a ) extends self-referential code evolution by fusing the task-solver and the meta-improver into a single editable program, so the self-modification mechanism is itself modifiable; tasks are externally fixed and the foundation model is frozen. These directions are complementary to SPADE : they address how agents use accumulated experience or rewrite their own scaffolding at the meta-level, while SPADE addresses how to generate the training environments that produce that experience in the first place via RL. HyperAgents explicitly leaves co-evolving the task distribution as future work, which SPADE addresses directly. The approaches are composable: SPADE ’s adaptive environment generation could populate the experience streams that ALMA-style memory systems and MemRL-style runtime RL operate over.

## Appendix I Extended Quantitative Analysis [back to contents]

Throughout, environment text is embedded with SBERT ( all-MiniLM-L6-v2 ) ( Reimers and Gurevych, 2019 ) , and every embedding-based result is replicated under a second, purely lexical embedding (TF-IDF with LSA-128), following the multi-embedding robustness protocol of Abdulhai et al. (2026) ; conclusions are identical under both. Step-level training dynamics use the matched 30B-A3B runs (full SPADE is the resume-stitched 0–399 step run; ablations end earlier), with curves EMA-smoothed over real logged values, expanding on Section 6.2 .

### I.1 Games

The games-setting analyses below expand the environment-quality and diversity signals summarized in the main text.

#### I.1.1 Environment Quality