Toward Training Superintelligent Software Agents through Self-Play SWE-RL — part 4/4
Cleaned-text code points [46654, 56747). Bibliography omitted here; source.md includes it.
2. Do not let the challenger diverge much from the initial instruction-based strategy and ensure that the self-play does not reach the tunnel-vision strategy. The challenger can still learn skills like passing the consistency checks and some calibration of question difficulty.
3. Do not attempt to improve on natural language skills using self-play without continued grounding on current human natural language. Gaining technical skills and code world model knowledge from self-play is desirable and plausible.
## Appendix B Opposing Incentives in the Reward Design
The solve rate s s in Equation 1 is computed over all solver attempts (e.g., 8 attempts per bug). Each individual solver attempt receives the binary reward r solve ∈ { + 1 , − 1 } r_{\text{solve}}\in\{+1,-1\} based solely on whether it succeeds. Given a bug with solve rate s s , the expected solver reward is 𝔼 [ r solve ] = s − ( 1 − s ) = 2 s − 1 \mathbb{E}[r_{\text{solve}}]=s-(1-s)=2s-1 . For valid bugs with ideal difficulty ( 0 < s < 1 0<s<1 ), the injection reward simplifies to r inject = 1 − ( 1 + α ) s r_{\text{inject}}=1-(1+\alpha)s . Consequently, the solver benefits from higher s s (easier bugs increase expected reward), while the injector benefits from lower s s (harder bugs increase injection reward), though bugs that are too hard ( s = 0 s=0 ) are penalized to prevent unsolvable proposals. This adversarial pressure pushes the injector to propose bugs near the frontier of the solver’s current capability. Note the optimal target difficulty depends on the sample size, explained in Appendix A .
## Appendix C Experimental Details
#### Training details.
We implement SSR on top of the async CWM-RL infrastructure ( FAIR CodeGen team et al., 2025 ) and adopt the same RL algorithm, with some hyperparameter modifications. We train the models on NVIDIA H100 SXM 80G GPUs, with a standard configuration of 512 GPUs for a single training run, with 64 GPUs for training and 448 GPUs for rollouts. Inspired by ScaleRL ( Devvrit et al., 2026 ) and MiniRL ( Zheng et al., 2025 ) , we incorporate a “large-batch” and “small-policy-staleness” hyperparameter setup. By default, we employ a 131,072 maximum context window size for generation, pack training sequences by a maximum of 131,072 tokens, use a global batch size of 16M tokens, achieved through 16 gradient accumulation steps per optimizer step, and a typical group size of 8 for acceptable rollout latency, discard rollouts with more than 8 off-policy steps, and train the models (including baseline RL, SSR, and all ablation runs) for 150 global steps, roughly 2.5B tokens, with 30 steps of learning rate warmup toward 3 × 10 − 6 3\times 10^{-6} .
Our implementation employs the same tool-based scaffold as CWM with minor prompt adjustments, incorporating Bash and a search-replace editor. All environment images used in our experiments are identical to those in CWM.
#### Evaluation noise.
While our method demonstrates consistent self-improvements across the training steps, we acknowledge the presence of evaluation noise, typically around 2% paired standard error on SWE-bench Verified. Readers can refer to Eval Arena ( Wang et al., 2024 ) for a detailed discussion.
## Appendix D Prompt Templates
### D.1 Bug Injection
### D.2 Bug Solving
For bug-solving with our solver agent, we instantiate the issue description with the following fixed prompt template:
We then use the following prompt for general issue solving, including both the real issues from evaluation and the fixed-template issues above:
## Appendix E Discussion
### E.1 Theoretical Analysis
With a focus on the challenger (bug-injector), we analyze the behavior of reward designs such as Equation 1 and the theoretical optimal play of the challenger-solver game in Appendix A . We show that the challenger has several dominant strategies that can stop self-play from progressing and then discuss practical mitigations that may address these problems, some of which were adopted by this work. Our analysis help clarify what we can expect in such self-play and what must be avoided.
### E.2 Limitations
While Self-play SWE-RL demonstrates promising results toward training superintelligent software agents, it is still an early step and several limitations warrant discussion: • No hidden oracles : Providing the complete oracle (i.e. the tests) in the task specification prompt may enable agents to develop reward-hacking behaviors (e.g., overfitting the specific set of tests) rather than genuine bug-fixing capabilities. Although this is not observed in this paper, future iterations should explore enriching the bug artifact with public and private test splits.
• Unit tests for verification : The current framework exclusively uses unit tests as the verification oracle, which represents only a subset of real-world software engineering activities. A higher-level abstraction, such as a goal, emphasized by CodeClash ( Yang et al., 2026b ) , may serve as a more scalable way to verify software correctness.
• Lack of model variants and role separation: Our experiments use a single model configuration for both roles. Future work should explore larger Mixture-of-Experts (MoE) models to understand how capacity and architecture affect self-play learning, and consider separate policies for each role.
### E.3 Unsuccessful Attempts
We share our setbacks here for insight, but this does not imply they cannot work in the future: • Synthesizing natural language issues in self-play : Our approach focuses on formal test specifications instead of natural language issues. While this design minimizes data assumptions and proves effective for learning, our initial attempts failed to reliably generate high-quality and unambiguous issue descriptions. The generated issues tend to copy test patches, are logically incoherent, and collapse to identical patterns. We attribute this to limited natural language capabilities of our 32B base model (CWM) and the opaque reward signals that fail to promote quality or diversity.
• Repository-specialized training . Real-world development often involves sustained work within specific codebases where deep contextual knowledge provides significant advantages. We explored training repository-specialized agents by performing SSR on just 23 repo images from SWE-bench Verified and SWE-Bench Pro, preventing leakage by selecting commits prior to any evaluation instances. However, this did not outperform training on more non-overlapping repos, likely because 23 repos provide insufficient diversity in problem-solving patterns.
• Stable training at scale : Despite following ( Devvrit et al., 2026 ; Zheng et al., 2025 ) , we still observed training instability that prevents SSR from further scaling, manifesting as gibberish outputs. This instability may stem from a combination of suboptimal hyperparameter settings, intrinsic challenges of long-horizon rollouts, model pretraining dynamics, and fundamental properties of self-play learning. Addressing this instability is critical to understanding how self-play scales and to unlocking its full potential.
### E.4 Future Work
Besides addressing the aforementioned limitations, our work opens several promising research directions for advancing self-play learning and superintelligence in software engineering agents: • Distribution control with seeding: The current approach lacks explicit control over bug injection locations, which can lead to duplicate bugs or distribution bias when sampling multiple times from the same repository. To improve diversity, future work could adopt seeding techniques from Magicoder ( Wei et al., 2024 ) , providing the bug-injection agent with contextual information such as target code snippets or specific files to guide the generation process toward diversity.
• Synthesizing complex multi-step software tasks: While our approach handles repository-level bug fixing, many real-world tasks require complex, multi-step workflows, such as major version migrations or building new software stacks from scratch. Our concept of higher-order bugs is a first step, introducing layered dependencies between bug injection and repair. Enabling such tasks will require new scaffolding designs, such as multi-context rollouts spanning interdependent sessions and automated context compaction ( Anthropic, 2025 ; OpenAI, 2025 ) .
• Efficient training paradigms for long-horizon software agents: Real-world software projects, such as building a production-grade RL codebase, pose unique challenges for agent training: they span months of iterative development, involve thousands of interdependent decisions, and cannot be fully validated by unit tests alone. Current outcome-based RL is inefficient for such tasks because sparse terminal rewards provide little signal when most trajectories fail, making credit assignment across thousands of steps intractable. Addressing this requires new paradigms that exploit the asymmetry between generation and verification ( Wei, 2025 ) for self-verification while delivering dense and structured feedback beyond scalar rewards.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .