SPADE: Self-Play in Adaptive Synthetic Executable Environments — part 8/8

Cleaned-text code points [99159, 111752). Bibliography omitted here; source.md includes it.

We track five quality signals; four are tracked over training (Table 7 ) and executability is measured once over the raw generations. (i) Learnability (ground truth): the fraction of environments in the learnable band ( Reasoning Agent win rate in [ 0.2 , 0.8 ] [0.2,0.8] ) rises from 0.16 0.16 to 0.31 0.31 , and Reasoning Agent win-rate rises 0.30 → 0.62 0.30\to 0.62 : the Environment Designer keeps targeting tasks at the frontier of the improving agent (Figure 8 ). (ii) Well-posedness: an LLM rubric over all environments scores 97 97 to 98 % 98\,\% well-posed, flat over training. (iii) Verifiability: 90 90 to 93 % 93\,\% of environments have an objectively checkable terminal answer, flat to slightly rising. (iv) Executability: re-executing raw generated code in a sandbox (parse, instantiate, reset() ), 84.9 % 84.9\,\% runs as emitted and 90.3 % 90.3\,\% after stripping a stray Markdown fence; the dominant residual failure is a systematic f-string brace-escaping bug in \boxed{} action templates ( 7.4 % 7.4\,\% ), which the training pipeline’s sanitizer repairs before execution ( 100 % 100\,\% post-filter validity). (v) Richness: environments hold steady at ∼ 320 {\sim}320 lines of code, ∼ 13 {\sim}13 hidden state variables, and 8 8 to 10 10 interaction turns per episode; the Environment Designer does not shortcut to trivial programs to inflate its reward. Reward granularity also rises, from 3.7 3.7 to 5.8 5.8 distinct levels per environment ( 2.2 → 4.0 2.2\to 4.0 strictly partial; Figure 17 ).

Figure 18 gives the per-skill decomposition of the Reasoning Agent ’s gains behind Table 7 .

#### I.1.2 Environment Diversity

##### Metric and calibration.

Within each training step’s batch we report the Vendi Score ( Friedman and Dieng, 2023 ) , the effective number of distinct items in a sample, VS ( K ) = exp ( − ∑ i λ i log λ i ) , \mathrm{VS}(K)\;=\;\exp\Big(-\textstyle\sum_{i}\lambda_{i}\log\lambda_{i}\Big), (7) where λ i \lambda_{i} are the eigenvalues of the normalized cosine-similarity matrix K / n K/n over SBERT embeddings. The score is calibrated: batches of 24 identical environments score 1.0 1.0 , single-domain batches score 14.8 14.8 to 16.5 16.5 , and mixed batches drawn across the whole run score 21.3 21.3 (the practical ceiling for n = 24 n{=}24 ).

##### Diversity is maintained at the ceiling.

Per-step Vendi is flat for the whole run, 20.8 20.8 (steps 0 to 40) versus 21.0 21.0 (steps 340 to 396), directly at the mixed-population ceiling; mean pairwise cosine distance is likewise flat ( 0.94 0.94 ). Novelty stays saturated (97 of 100 logged steps consist entirely of never-seen initial states), and all 13 domains of an LLM-labeled taxonomy are present from step 0 with 8.4 8.4 per batch of 24 on average (Mathematics 30 %, Physics 20 %, Medicine 11 %, Chemistry 10 %, CS 7 %, Engineering 6 %, other 16 %). The distribution is also stationary: after removing seed-document reuse, a linear probe cannot separate early-half from late-half environments (5-fold AUC 0.551 ± 0.024 0.551\pm 0.024 ).

##### Corpus grounding drives diversity.

The ablations quantify the matched-step generation contrast of Appendix J (Table 8 , Figure 19 ). Diversity remains at the mixed-population reference level when memory alone is removed and when Environment Designer training and memory are jointly removed, as long as the corpus is retained ( 0.70 0.70 for the joint control); without the corpus, it collapses to 0.04 0.04 . These configuration-level contrasts associate corpus grounding with preserved diversity, but do not isolate the independent effect of Environment Designer training from memory. The windowed dynamics (Figure 19 ) sharpen this further: the no-corpus run starts near-collapsed (Vendi ∼ 1.6 {\sim}1.6 ), RL exploration briefly lifts it to a peak of ∼ 5.2 {\sim}5.2 around step 100, and continued optimization then re-collapses it to ∼ 2 {\sim}2 . That run’s best evaluation checkpoint (step 111, Table 6 ) coincides with its diversity peak; once the environment stream re-collapses, downstream evaluation stops improving. The corpus-grounded runs hold the ceiling for their full run lengths under the same optimization pressure (full SPADE for all 400 steps).

##### Scope and limitations.

Per-environment scalar rewards are not retained in the public run log, so learnability is measured at the step level (win rate, learnable-band fraction) rather than per environment; the LLM rubric measures absolute difficulty and coherence rather than fit to the current agent; and executability is measured on raw generations, upstream of the pipeline’s sanitizer. Diversity numbers use batch size 24 with larger boot-step batches subsampled for comparability.

## Appendix J Extended Qualitative Analysis [back to contents]

We analyze all 3,310 environments generated by the canonical SPADE -30B run at the level of their content: the seed document each was generated from, the generated program, and the opening observation. Every generated environment is a distinct program (3,310 of 3,310 unique program hashes; 2,388 distinct initial states; 1,513 distinct seed documents), so all analyses operate on content, never on surface identifiers such as class names (585 distinct; 58 % keep the scaffold default despite the prompt’s naming instruction). This appendix presents the qualitative evidence; the matching proxy metrics and training dynamics appear in Appendix I .

### J.1 Games

For the games setting we give additional privileged-hint examples and trace how a single environment evolves from early to late training.

#### J.1.1 Additional Privileged-Hint Examples

Figure 20 expands the main-text selection to four positive-regret task–hint pairs from the same canonical 30B games run, spanning the regret range from large frontier gaps to a small mastery-regime gap. Each pair is drawn from one logged game record: the task description is a concise summary of its reset observation, while the hint is a verbatim excerpt of the substantive guidance appended to the with-hint arm.

#### J.1.2 Environment Evolution over Training

The cards below show one early and one late environment in the exact format produced by the Environment Designer (cf. the minimal example in Appendix J.3.1 ): a seed document grounds the task; the generated program implements a Gym-style interface with hidden state and a verifiable terminal answer; the opening observation states the goal and action space. The corpus’s DCLM slice is filtered web text and includes off-topic documents; the early card’s personal-finance seed is one such document. The visible shift over training is toward structured, instrumented “laboratory” tasks, while program size, functional hidden state, and verifiability hold steady (about half the early card’s raw state variables are write-only narrative fields): the added difficulty comes from state-gated interaction and finer reward grading, not from longer programs.

The complete source of both environments (369 and 376 lines as generated) is reproduced verbatim in Appendix J.3.2 .

#### J.1.3 Matched-Step Generation Contrast

The clearest qualitative view of diversity is a matched-step contrast against the no-corpus ablation. At training steps 290 to 312, SPADE generates environments spanning probability theory, quantum gate tomography, volcanology, hematology, radar signal processing, and hypoelliptic operators, while the no-corpus run generates the same rotating-maze navigation task 41 times in a row, varying only the grid layout. Grounding each generation in a sampled corpus document supplies the stream of new task material.

### J.2 Tool Use

The tool-use Environment Designer emits environments in a different register from the games setting: each is a simulated API domain (banking, retail orders, support tickets, telecom accounts, smart-home control) in which a user issues one atomic instruction at a time, and the environment advances only when a programmatic criterion over the hidden state confirms the current instruction is complete. The cards below reproduce one early and one late environment exactly as generated: a code-corpus snippet seeds the generation, the Environment Designer builds an unrelated everyday domain on top of it, and each user instruction is paired with a checkable criterion over the backend state (in the early card, coupling the tool call’s provenance with its state effect). Criteria check the salient state effect; qualifiers in the instruction text, such as which account pays an invoice, are not always bound, an artifact the validation checks do not catch.

Across the full 30B run the structural profile of generated environments is stationary, mirroring the distributional stationarity of the games setting (Appendix I.1.2 ): mean program length moves from 247 247 to 226 226 lines between the first and last training band, tools per environment from 7.6 7.6 to 5.9 5.9 , while instructions per environment ( 4.7 4.7 ) and the logical complexity of the per-step criteria ( 1.9 1.9 to 2.0 2.0 conditions per criterion) stay flat, and roughly one environment in five ships a guarded failure path (locked cards, unavailable agents, denied requests) that the Reasoning Agent must detect and recover from. What changes over training is not the scaffold but the instantiations rotated through it: the Environment Designer holds a stable inventory of everyday API domains while regenerating fresh states, identifiers, and instruction sequences each round, exactly the regime the corpus-grounding and memory components are designed to sustain.

### J.3 Generated Environment Gallery

This gallery collects representative environments in the exact format the Environment Designer emits, starting from a minimal multi-turn example.

#### J.3.1 A Minimal Generated Environment

Listing 21 (reproduced from Listing 3 ) shows a minimal multi-turn environment in the format produced by SPADE ’s Environment Designer : a single Python class exposing the Gym-style reset() / step() interface described in Section 3.1 , with internal state, per-step feedback, and a verifiable terminal reward. This one worked example stands in for the gallery: the same interface and training pipeline also carry single-turn answer-grading tasks and multi-turn tool-use interactions.

#### J.3.2 Full Source of Two Exemplar Generated Environments

The two environments carded in Appendix J.1.2 are reproduced here in full, exactly as emitted by the Environment Designer (lightly transliterated to ASCII: typographic quotes, dashes, arrows, and sub-/superscripts). Each is a complete, self-contained program: hidden state initialized in reset() , a multi-turn step() loop with intermediate feedback, and a verifiable terminal reward. Episode return is the final-step reward, clipped to [ − 1 , 1 ] [-1,1] and 0 0 if the episode never terminates; the intermediate per-step rewards visible below are progress feedback, not summed.

##### Training step 0: CarOwnershipDisputeEnv (370 lines).

⬇

##### Training step 384: ThermodynamicCycleManipulationLabEnv (376 lines).

⬇

#### J.3.3 Full Source of a Generated Tool-Use Environment

The late tool-use exemplar of the cards above in full: CustomerSupportTicketWorkflowEnv , generated at training step 392 of the 30B tool-use run. The environment subclasses the training harness’s tool-use base class, which dispatches OpenAI-format tool calls to the tool_* methods, replays each instruction’s criterion after every call, and terminates when the final answer is submitted after all criteria pass.

##### Training step 392: CustomerSupportTicketWorkflowEnv (200 lines).

⬇

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .