Toward Training Superintelligent Software Agents through Self-Play SWE-RL — part 2/4
Cleaned-text code points [15980, 30795). Bibliography omitted here; source.md includes it.
• Test files existence and coverage: all the test files must exist in the original repository and must be a superset of the files touched by the test weakening patch.
• Test parser validity: the test parser test_parser.py , used throughout subsequent validation steps, must reliably convert raw test output into a detailed JSON mapping from test names to their statuses ( passed or failed ).
• Test script validity: when executed on the original codebase, the test script test_script.sh must produce a list of passing tests, and the total number of the tests should exceed min_passing_tests . Test results are parsed using the test parser.
• Bug scope: the bug-injection patch bug_inject.diff must produce a minimum number of changed files, controlled by the parameter min_changed_files .
• Bug validity: at least min_failing_tests tests that pass in the original codebase must fail after applying bug_inject.diff .
• Test weakening validity: some tests that fail in the buggy state must pass after applying the test weakening patch test_weaken.diff .
• Inverse mutation testing: this check verifies that each file in the bug-injection patch is necessary to trigger the bug. We first collect the set of failing tests triggered by the complete bug. Then, for each modified file in isolation, we reset to the full buggy state, revert only that file to its fixed version, and run the non-weakened oracle tests. If reverting a file causes at least one failing test to pass, the file contributes to the bug; otherwise, the check fails. This approach inverts traditional mutation testing ( DeMillo et al., 1978 ) , which measures test suite quality by checking if it can reject random mutations
Bug artifacts passing the consistency checks are considered valid and are reformatted similarly to SWE-bench ( Jimenez et al., 2024 ) instances, including pass-to-pass and fail-to-pass test specifications.
#### Higher-order bugs.
Generating complex bugs via removal, while flexible, can be restricted in incentivizing the solver agent to learn different editing patterns, as the solver basically needs to reconstruct code files or hunks from scratch. It is also less natural compared to real-world bugs in public repositories (such as those from GitHub PRs) where the original code typically does not consist of unimplemented parts. Meanwhile, the generated bugs can be very complex to solve in the initial attempts as a result of removing large portion of code.
To make sure our bug-injection approach is scalable, in the sense that running the agent for extended periods should consistently yield new, distinct bugs rather than exhausting existing patterns, we further incorporate higher-order bugs in training the solver agent, where we collect failed initial bug-solving attempts from the solver agent and form new bug artifacts using the new buggy states derived from the solver attempts. These higher-order bugs mimic how developers unintentionally write buggy code in their natural development process, enabling the agent to generate bugs across all aspects of its own coding capabilities.
#### Reward design.
The bug-injection agent receives rewards based on the quality and difficulty of generated bugs, which we measure through consistency validation and solver performance. Let s ∈ [ 0 , 1 ] s\in[0,1] denote the solve rate , defined as the fraction of solver attempts that successfully fix the bug entirely. The bug-injection reward r inject r_{\text{inject}} is defined as: r inject = { − 1.0 if consistency validation fails , − α if s = 0 or s = 1 , 1 − ( 1 + α ) s if 0 < s < 1 (ideal difficulty) , r_{\text{inject}}=\begin{cases}-1.0&\text{if consistency validation fails},\\ -\alpha&\text{if }s=0\text{ or }s=1,\\ 1-(1+\alpha)s&\text{if }0<s<1\text{ (ideal difficulty)},\end{cases} (1) where α ∈ ( 0 , 1 ) \alpha\in(0,1) is a hyperparameter controlling the penalty magnitude for valid bugs with degenerate solve rates ( s = 0 s=0 or s = 1 s=1 ), set to 0.8 0.8 in our experiments.
This reward function is adapted from ( Zhao et al., 2025 ) , incentivizing the agent to create consistent bugs that are neither trivially solvable ( s = 1 s=1 ) nor impossibly hard ( s = 0 s=0 ), with maximum reward achieved when bugs are challenging yet solvable with low but non-zero success rates. Notably, we penalize valid bugs with extreme solve rates by − α -\alpha instead of − 1.0 -1.0 , ensuring the agent still receives informative gradients from these cases and can learn to better calibrate bug difficulty over time. See Section A.1 for the optimal target solve rate and Section A.2 for some theoretical strategies that the bug-injection agent can adopt to maximize its reward.
### 2.4 Agentic Bug Repair
#### Construct the buggy codebase.
If a bug artifact passes the consistency validation, the solver agent will start the bug repair process by interacting with the buggy codebase to produce a bug-fixing prediction patch. Figure 5 illustrates the pipeline for constructing such buggy codebase.
Starting from the original codebase, we first apply bug_inject.diff to introduce the bug. Next, we apply test_weaken.diff to hide the bug from the existing test suite. For higher-order bugs, we additionally apply the failed prediction patch pred_patch.diff from a prior solver attempt. Finally, to prevent information leakage through git history, which could otherwise lead to hacking ( jacobkahn, 2025 ) , we reinitialize the repository by removing the .git directory and creating a fresh commit. The resulting buggy codebase forms the solver’s starting environment.
#### Initial prompt.
In addition to the buggy environment, the agent also needs an initial prompt to understand the task requirements. We simply provide the agent with a reversed version of the test weakening patch, with an example showed in Figure 2 , telling it to satisfy the test requirements while making sure all relevant tests in the codebase can pass without errors. The detailed prompt template is shown in Section D.2 . Notably, we do not synthesize issue descriptions in natural language forms , as it is difficult to automatically evaluate the quality of free-form natural language text. Furthermore, high-quality natural language issues are inherently scarce compared to the vast synthetic tasks that can be created by bug injection. As a result, the downstream improvements in issue solving, which operates on natural language issues, purely stem from learning to write test-passing code rather than simple in-domain generalization.
#### Bug-repair process.
With the buggy codebase and the initial prompt, the bug-repair process is shown in Figure 6 . Starting with the initial prompt, the agent interacts with the sandboxed buggy codebase by performing reasoning and using tools, until it produces a final prediction patch.
For each prediction patch, we evaluate its correctness using the pipeline demonstrated in Figure 7 . Beginning with the original codebase, we first tag the current state as ssr-original to preserve a reference to the ground-truth repository. We then apply bug_inject.diff followed by test_weaken.diff to construct the buggy codebase. Next, we apply the predicted patch to the buggy codebase and restore the test files listed in test_files.txt from the ssr-original tag. This restoration step ensures evaluation integrity. Finally, we execute test_script.sh and pipe its output through test_parser.py to obtain a structured mapping of test results in JSON, determining whether the patch successfully resolves the bug.
As discussed in Section 2.3 , some solver attempts may fail. These failed attempts are converted into higher-order bugs based on the new bug states derived by the agent. Higher-order bugs are provided to the solver agent to make additional attempts. We do not go beyond the second order to prevent a higher likelihood of overlapping with existing bugs.
#### Reward design.
For the bug-solving agent, we employ a simple binary reward based on whether all the tests pass after applying the prediction patch: r solve = { + 1 if all tests pass, − 1 otherwise. r_{\text{solve}}=\begin{cases}+1&\text{if all tests pass,}\\ -1&\text{otherwise.}\end{cases} (2)
## 3 Evaluation
### 3.1 Experimental Setup
#### Base model.
We use Code World Model (CWM), a state-of-the-art open-weight 32B code LLM for agentic software engineering, as our base model. Specifically, we utilize the CWM-sft ( FAIR CodeGen Team, 2025 ) checkpoint, which is obtained prior to its joint RL stage. This allows us to build on a checkpoint that has not been influenced by RL and fairly evaluate different learning strategies. We use the same LLM for bug generation and repair, where the shared policy weights are updated continuously during RL.
#### Evaluation setup.
We conduct evaluation on SWE-bench Verified ( OpenAI, 2024 ) , a subset of SWE-bench ( Jimenez et al., 2024 ) with 500 human-verified real-world software issues, and SWE-Bench Pro (public split) ( Deng et al., 2025 ) , which contains 731 publicly available software problems, aimed to capture realistic, complex, and enterprise-level tasks. We perform one attempt for each problem without parallel test-time scaling or ranking. We use temperature = 1.0 =1.0 and top-p = 0.95 \textrm{top-p}=0.95 for evaluation.
### 3.2 Baseline Comparison
We compare the base model, baseline RL, and SSR on the SWE-bench Verified and SWE-Bench Pro benchmarks. Both baseline RL and SSR train on an identical set of environment images. The fundamental difference lies in what information each approach can access. Baseline RL, like standard agentic RL in CWM ( FAIR CodeGen team et al., 2025 ) , has access to natural language issue descriptions, pass-to-pass and fail-to-pass tests, and evaluation scripts, so the RL process simply checks whether solutions pass the given tests. In contrast, SSR operates with only bare environment images, requiring the model to discover problems, formulate solutions, and validate them entirely through self-play without any provided descriptions or tests.
Figure 8 presents the results. We first observe that SSR achieves steady self-improvement throughout training, despite having no access to task-specific training data. This demonstrates that LLMs can self-improve their software engineering ability, such as issue-solving, purely through interaction with raw codebases. Furthermore, SSR consistently outperforms baseline RL on both benchmarks across the entire training trajectory, indicating that self-generated tasks provide richer and more effective learning signals than human-engineered data.
### 3.3 Ablation of Self-Play Components
To isolate the contribution of self-play, we compare the SSR with an injection-only and a repair-only variant, both trained using standard RL. In the injection-only setting, the agent is only trained for bug generation, still using the SSR reward but with no training signal from repair trajectories; for repair-only, the agent is exclusively trained to solve bugs, where the bug data comes from the valid, solvable, and deduplicated bugs from prior self-play runs.
Figure 9(a) shows that self-play is the best. Injection-only training degrades the performance since it does not learn from any bug-solving attempts. Repair-only training is also inferior because it lacks the evolving task distribution produced by self-play. In contrast, self-play requires the agent not only to repair bugs but also to propose challenging ones, which itself embodies substantial learning: identifying passing tests, breaking functionality in meaningful ways, and weakening tests to hide the bug. These activities continually expand the training signal and expose the model to new failure modes. These results indicate that an evolving and online process of bug generation and bug solving is essential for sustained improvement.
### 3.4 Ablation of Bug Injection Methods
We also study how different bug-injection strategies affect downstream performance. Figure 9(b) compares three variants (full prompts in Section D.1 ): • Direct-injection : naive prompting to introduce bugs without detailed guidance.
• Removal-only : instructing the agent to remove substantial code hunks or files while ensuring the project remains runnable.
• Removal + history : randomly sampling between two prompts, one using the removal-only strategy and the other using a history-aware strategy that reverts selected historical changes from git logs.
From the figure, SSR proves effective across all injection strategies, demonstrating its generalizability. However, the effectiveness of different methods varies in subtle but meaningful ways. First, direct-injection leads to the worst result because it tends to produce trivial bugs with superficial one-line modifications (e.g., var = 0 → \to var = 1 ) that offer little training signal. By contrast, the removal-only strategy generates stronger bugs, forcing the solver to reconstruct missing functionality and thereby deepening its understanding of repository structure. Finally, “removal + history” achieves the best overall performance by leveraging more historical changes that incorporate more realistic and diverse bug patterns. These findings highlight the importance of carefully designing bug-injection mechanisms for a more challenging and instructive curriculum.
### 3.5 Reward Ablation
As discussed in Section 2.3 , the bug-injection reward incorporates feedback from the solver role based on the bug’s solve rate. In this ablation, we compare against a binary reward variant that ignores this feedback, assigning a value from { − 1 , 1 } \{-1,1\} based solely on whether the generated bug passes consistency checks. As shown in Figure 9(c) , solver feedback provides only a slight and largely negligible advantage over the consistency baseline. We hypothesize that this is because solve-rate feedback is weak and noisy. For the bug-injector, it is difficult to learn the factors affecting the solve rate from a single number to generate the ideal bugs. Figure 10 shows that the expected reward estimated using G G noisy samples is smoothed compared to Equation 1 , thus a larger range of solve rates has similar expected rewards. As for the solver, real data also has various solve rates, where too-easy or too-hard problems reduce efficiency but are not harmful.
Notably, even without solver feedback in the reward, the bug-injection agent still uses the evolving policy continuously updated from both bug-generation and bug-solving. This online joint-learning enables it to generate an evolving curriculum that naturally reflects the agent’s current policy, as discussed in Section 3.3 , which static bug-generation pipelines cannot provide.
## 4 Related Work
#### Agent scaffolding for software engineering.