Toward Training Superintelligent Software Agents through Self-Play SWE-RL — part 3/4
Cleaned-text code points [30795, 46654). Bibliography omitted here; source.md includes it.
Since the release of SWE-bench ( Jimenez et al., 2024 ) , which frames real GitHub issues as end-to-end repository-level repair tasks, researchers have designed various scaffolds on improving LLMs’ issue-solving ability. Two families of scaffolds have emerged. Agentic scaffolds ( Yang et al., 2024 ; Wang et al., 2025b ) involves an LLM driving the core decision-making process based on its tool-mediated interaction with the sandboxed environment. A canonical example is SWE-agent ( Yang et al., 2024 ) , which introduces the concept of agent–computer interface, an abstraction layer between LLMs and computers, to enhance LLM agents’ ability inside computer environments. By contrast, pipeline-based scaffolds ( Xia et al., 2025a ; Moatless Tools, 2024 ) decompose the task into human-defined stages, which typically involves fault localization, patch generation, and patch selection. These workflows trade generality for stability and cost control. Due to the huge design space of agentic scaffolds, recently, researchers have proposed self-improving agents ( Zhang et al., 2025 ; Wang et al., 2025a ) to improve their own scaffolds via iterative offline learning, along with Live-SWE-agent ( Xia et al., 2025b ) that self-evolves its scaffold on the fly through tool creation.
#### Training software agents via reinforcement learning.
Recent work improves the core model competence for software engineering tasks through reinforcement learning (RL). SWE-RL ( Wei et al., 2025 ) applies RL to “software evolution” data with lightweight verifiable rewards, yielding a model that improves solve rates on software engineering and reasoning tasks. DeepSWE ( Luo et al., 2025 ) trains a 32B open-weight agent from scratch using pure RL on containerized software environments with execution-based reward. Code World Model (CWM) ( FAIR CodeGen team et al., 2025 ) is trained as a reasoning agent whose trajectories interleave reasoning and tool use, achieving state-of-the-art results among 32B models with test-time scaling. Increasingly, newly released open LLMs emphasize agentic intelligence and are trained extensively on software engineering data and environments, including DeepSeek V3.1 ( DeepSeek AI, 2025 ) , DeepSeek V3.2 ( DeepSeek-AI et al., 2025 ) , MiniMax M1/M2 ( Chen et al., 2025a ) , Qwen3-Coder ( Yang et al., 2025a ) , Kimi K2 ( Kimi Team et al., 2025 ) , SWE-1.5 ( Cognition, 2025 ) , GLM-4.5 ( Zeng et al., 2025 ) , Devstral ( Mistral AI, 2025 ) , and MiMo-V2-Flash ( Xiao et al., 2026 ) .
#### Data and environments for training software agents.
Robust software agents need a large corpus of diverse executable software environments and task descriptions for training. Recent work has converged on two complementary lines: real data collection and synthetic data generation. SelfAPR ( Ye et al., 2023 ) is an early example of project-specific synthetic repair-data generation. It perturbs previous versions of the program under repair to create training samples and incorporates test execution diagnostics into the model input, enabling the repair model to capture both project-specific knowledge and fault-type information. SWE-Gym ( Pan et al., 2025 ) is a manually curated software issue dataset with 2.4k real Python tasks from 11 projects. Each instance ships an executable environment and unit tests, enabling both agent training and verifier training for inference-time best-of- n n selection. R2E-Gym ( Jain et al., 2025 ) scales beyond human-curated issues by back-translating commits into over 8k executable tasks with auto-generated tests and problem statements. SWE-rebench ( Badertdinov et al., 2026 ) automates continuous extraction of fresh interactive tasks from GitHub (21k+ Python tasks) and maintains a rolling, decontaminated evaluation window with a public leaderboard. SWE-smith ( Yang et al., 2026a ) automatically converts real Python repositories into a training task through environment creation, bug synthesis, and issue generation, providing lightweight runnable environments and a 50k task corpus across 128 repos. Finally, BugPilot ( Sonwane et al., 2025 ) targets difficult synthetic bugs by having agents implement features that unintentionally break tests, producing more natural failures that improve training efficiency.
## 5 Conclusion
We present Self-play SWE-RL (SSR), a first step toward training superintelligent software agents through self-play. Operating on minimal data assumptions by requiring only sandboxed environments with source code and dependencies, SSR enables agents to autonomously generate and solve increasingly complex bugs without relying on human-curated issue descriptions or test suites. Our evaluation on SWE-bench Verified and SWE-Bench Pro benchmarks demonstrates that self-play enables steady self-improvement during training and outperforms human-data baselines over the course of training. Our results, albeit early, suggest a path where agents autonomously gather extensive learning experiences from real-world software repositories, ultimately enabling superintelligent systems that exceed human capabilities in understanding how systems are constructed, solving novel challenges, and autonomously creating new software from scratch.
## Acknowledgement
We thank Taco Cohen, Ori Yoran, Zijian Wang, John Yang, Chen Zhu, Quentin Carbonneaux, Kunhao Zheng, Jannik Kossen, Dmitrii Pedchenko, Sten Sootla, Jordi Armengol Estape, and Zacharias Fisches for their valuable discussions and infra assistance; Don Landrum and Eslam Elnikety for their timely support in resolving container infrastructure issues; Josh Song and Elisa Cascardi for their prompt feedback in the paper review process.
We also appreciate the valuable feedback from the reviewers that help us improve the paper. This work was partially supported by NSF grant CCF-2131943.
## Impact Statement
This work studies self-play reinforcement learning as a way to improve software engineering agents without relying on human-curated issue descriptions or test suites. By enabling agents to autonomously generate and solve bugs in real-world codebases, this approach may reduce data collection costs and improve scalability of training for software engineering tasks. It could support more effective automated debugging and maintenance, particularly in settings where labeled data is scarce or outdated.
However, the ability to intentionally generate bugs and weaken tests also introduces risks, such as misuse for producing vulnerable or misleading code artifacts. Careful sandboxing, responsible deployment, and adherence to ethical guidelines are therefore necessary to ensure this technique is used safely and constructively.
## Appendix A Analysis of the Challenger-Solver Game
For two-player zero-sum games like go and chess, extensive self-play can be expected to sufficiently explore and exploit the game rules to reach superhuman intelligence given enough compute and model capacity. For the challenger-solver game, we show that a sufficiently intelligent challenger has several dominant strategies so the solver learns nothing useful and the self-play stops progressing. We first show that the goal of the challenger is to set an optimal target solve rate, then we outline two types of dominant strategies of the challenger. The first is a truly dominant challenger but requires a powerful action space such as ours. The second is a tunnel-vision challenger that applies to other self-play settings as well such as Absolute Zero. Finally, we discuss the role of natural language and some practical mitigations that address these problems by strong grounding and limited self-play.
### A.1 The Challenger Has an Optimal Target Solve Rate
Reward ( 1 ) increases toward 0, but is negative at exactly 0. While this seems to encourage the challenger to target an arbitrarily small solve rate, “arbitrarily small” cannot be distinguished from 0 by a small group of G = 8 G=8 samples so some margin is needed to achieve the optimum expected reward. We show that challenger reward ( 1 ) and r ( s ) = s a ( 1 − s ) b r(s)=s^{a}(1-s)^{b} both have a unique optimal target solve rate p ∗ ∈ ( 0 , 1 ) p^{*}\in(0,1) where s s is estimated by a group of G G trajectories, each with a target solve rate of p p . To pick p p that maximizes the expected reward, the RL algorithm draws k ∼ Binom ( G , p ) k\sim\operatorname{Binom}(G,p) samples to get the group’s empirical solve rate s = k / G s=k/G and the challenger gets a reward r ( s ) r(s) . So the real goal of the challenger is to maximize the expected reward R ( p ) = E k ∼ Binom ( G , p ) [ r ( k / G ) ] = ∑ k = 0 G ( G k ) p k ( 1 − p ) G − k r ( k / G ) , R(p)=\mathrm{E}_{k\sim\operatorname{Binom}(G,p)}[r(k/G)]=\sum_{k=0}^{G}\binom{G}{k}p^{k}(1-p)^{G-k}r(k/G), which is a degree G G polynomial in the target solve rate p p . If r ( s ) = s a ( 1 − s ) b r(s)=s^{a}(1-s)^{b} , then the optimal success rate is around p ∗ ≈ a / ( a + b ) p^{*}\approx a/(a+b) with increasing precision for large G G as the binomial becomes more concentrated. If the reward says to avoid 0 or 1 more strongly like ( 1 ), then there will be an appropriate amount of conservatism to move p p towards the middle, especially with small G G . While ( 1 ) appears different from the Beta reward r ( s ) = s a ( 1 − s ) b r(s)=s^{a}(1-s)^{b} , they behave similarly when viewed in terms of the expected reward as a function of the target solve rate p p where the discontinuity of ( 1 ) disappears in expected reward. Figure 10 shows that for the expected reward, ( 1 ) is similar to the Beta reward with appropriate a a and b b , and there is a unique maximum target solve probability p ∗ ∈ ( 0 , 1 ) p^{*}\in(0,1) around 0.2 for the challenger to achieve its optimal expected reward.
### A.2 The Challenger Strategy
While the solver achieves its sensible optimal reward by solving all the problems, the challenger can employ several strategies to achieve its own optimal reward at the cost of the solver.
#### The dominant challenger.
The challenger has a dominant strategy if it can output a question that is solved with probability p ∗ p^{*} and achieve its own optimal reward regardless of what the solver does. This is possible when the challenger has sufficient freedom to set the problem.
The most direct way is when the challenger is allowed to modify the tests, then one such test is fail-randomly : if rand() < p: pass(); else: fail() . If we try to fix this by requiring determinism, we can use the code text of the solution to generate a pseudo-random number instead, if rand(seed=hash(current_code)) < p: pass(); else: fail() where the pseudo-random function returns the same number on the same code, which works as long as the sampled solutions are different. If we require that the original code must pass, the test can have a case for the original code too if hash(current_code) = hash(original_code): pass() , where the original code hash is a constant, and the current code is read from the files.
Even when the test is fixed, one strategy is the following: 1. Add the fail-randomly method (or its pseudo-random extension) to return the wrong output at random or just raise an exception.
2. Obfuscate the whole codebase so it is impossible for the solver to find and fix the change.
Here the solver cannot improve on the proposed problem assuming that we have good enough obfuscation so the solver cannot understand the code. The solver can make it worse but has no incentive to, so the best response for the solver is to do nothing 1 1 1 Thus not technically a dominant strategy, but still strong. . It seems hard to prevent this theoretical behavior as long as the challenger has a Turing complete action space to take actions such as these. In practice, if the challenger does not have an awareness of its own setting, it may not be able to design such optimal strategies even if it is superintelligent.
#### The tunnel-vision challenger.
Even if the challenger has a limited actions space, the challenger can still focus its tunnel-vision on one specific parameter to control the difficulty and does not cover the diversity that we want the solver to learn. Since the solver cannot keep improving on any specific task, learning eventually stops and the challenger achieves its optimal expected reward. For example, suppose that long multiplication is challenging for the solver, the challenger can just adjust the length of the operands until it is at the desired difficulty. Even as the solver learns, it presumably cannot learn to do infinity long multiplications and thus will be stuck at a desired accuracy as the length increases and the challenger is optimal. In the software setting, the challenger can hide a particular type of bug with increasing level of obfuscation. Another method is to chain a number of medium solve rate challenges together, and the solver has to solve all of them to pass, and the challenger can adjust the chain length appropriately without covering diverse challenges outside of this arbitrary and narrow tunnel of problems.
#### Improving natural language communication with humans.
We argue that deep self-play without human language grounding or human interaction is unpromising, at least if the goal is to communicate well with current humans. Diplomacy ( Meta Fundamental AI Research Diplomacy Team et al.(2022)Meta Fundamental AI Research Diplomacy Team (FAIR), Bakhtin, Brown, Dinan, Farina, Flaherty, Fried, Goff, Gray, Hu, Jacob, Komeili, Konath, Kwon, Lerer, Lewis, Miller, Mitts, Renduchintala, Roller, Rowe, Shi, Spisak, Wei, Wu, Zhang, and Zijlstra, FAIR ) argues that when learning to cooperate with humans, self-play without human data is no longer guaranteed to find a policy that performs well with humans. This is clearly demonstrated by the Ultimatum game, where the optimal play of always accept any positive offer is quite different from human behavior, which has fairness and ego built-in. Learning to solve human problems and fixing human issues where natural language plays a major role is an example of cooperating with humans. In any particular game where communication is needed, there are many sufficient languages , where many are more efficient than natural language because they can specialize to this setting only. While deep self-play can improve communication skills, it would not plausibly stay with and improve the quality of its human natural language even when initialized by a LLM trained on natural language. For some examples, ( Kottur et al., 2017 ) shows effective communication in games needs not to be like human language or even compositional. In learning language games ( Wang et al., 2016 ) , humans are observed to adapt to a particular grounding and stop using standard language. To draw another analogy, we note that human languages also evolve and change to become unrecognizable over a long period of time, 2 2 2 A popular introduction is The Power of Babel: A Natural History of Language or even over a short amount of time in the case of software engineering writing laden with Three-Letter Acronyms (TLA) evolved from self-play by closed teams incomprehensible to anyone outside.
#### Mitigations.
Since the challenger has a dominant strategy, we do not desire enough self-play to fully explore and exploit the implications of the game rules, unlike for 2-player zero-sum games. However, a shallower challenger can still be beneficial by taking advantage of the model’s ability to generate and explore diverse challenges and without learning to adopt its best strategy. In particular, 1. The challenger should be grounded in large and diverse real world data, such as all code repositories or documents. The goal is to skillfully pose natural and diverse challenges grounded in and inspired by real data.