Competitive Programming with Large Reasoning Models — part 2/3

Cleaned-text code points [15412, 31412). Bibliography omitted here; source.md includes it.

In addition to its significantly improved problem-solving capabilities, we observe that o3 demonstrates more insightful and deliberate chains of thought. The model not only writes and executes code to validate its solutions against public test cases, it also refines its approach based on these verifications. Figure 6 shows an advanced test-time strategy discovered by o3: for problems where verification is nontrivial, it often writes simple brute-force solutions — trading efficiency for correctness — then cross-checks the outputs against its more optimized algorithmic implementations. This self-imposed validation mechanism lets o3 catch potential errors and improve the reliability of its solutions.

### 4.2 IOI 2024 Benchmark

Although we competed in IOI 2024 using o1-ioi, we retrospectively evaluated a checkpoint of o3 on the same six IOI 2024 problems to compare performance under identical conditions. As with o1-ioi, we strictly adhered to the official IOI rules, which permit a maximum of 50 submissions per problem.

The o3 results on the IOI 2024 were produced by a later version of o3 than the CodeForces results, and included additional fresher training data. IOI 2024 occurred after the training cut-off for this model, and we additionally confirmed with search that the IOI test problems are not contaminated with the training set.

##### Sampling Approach.

Unlike o1-ioi, which sampled solutions separately for each subtask, we adopted a different approach when evaluating o3: sampling from a single prompt containing the original problem statement . Additionally, while o1-ioi generated 10K solutions per subtask, for o3 we sampled only 1K solutions per problem.

Selection strategies also differed between the two models. Whereas o1-ioi relied on a complex, human-defined test-time strategy ( 3.2 ) to select solutions, o3 followed a much simpler approach. Specifically, we selected the top 50 solutions with the highest test-time compute from 1,024 samples per problem. Despite this streamlined method, o3 produced robust solutions capable of covering many, if not all, subtasks — without the need for subtask-specific prompts, manual partitioning, or intricate submission strategies.

##### Results.

Figure 7 presents the final scores. The IOI scoring system is subtask-based, with a maximum total of 600 points in the 2024 contest. The gold medal threshold was approximately 360 points. Key results include:

• o1-ioi scored 213 points with 50 submissions, improving to 362.14 points with 10K submissions, just above the gold medal cutoff.

• o3 achieved 395.64 points, surpassing the gold threshold even under the 50-submission limit.

These results demonstrate that o3 outperforms o1-ioi without relying on IOI-specific, hand-crafted test-time strategies. Instead, the sophisticated test-time techniques that emerged during o3 training, such as generating brute-force solutions to verify outputs, served as a more than adequate replacement and eliminated the need for the hand-engineered clustering and selection pipelines required by o1-ioi.

Overall, the IOI 2024 findings confirm that large-scale RL training alone can achieve state-of-the-art coding and reasoning performance. By independently learning to generate, evaluate, and refine solutions, o3 surpasses o1-ioi without dependence on domain-specific heuristics or clustering-based methods.

## 5 Software Engineering Evaluations

We have demonstrated how reasoning significantly enhances LLM performance in competitive programming, where solving complex algorithmic challenges requires deep logical thinking. However, we also sought to evaluate the impact of reasoning on real-world coding tasks. To this end, we tested our models on two datasets: the HackerRank Astra 2 2 2 www.hackerrank.com/ai/astra dataset and SWE-bench verified 3 3 3 https://openai.com/index/introducing-swe-bench-verified/ [ 5 , 11 ] .

### 5.1 HackerRank Astra

The HackerRank Astra dataset is composed of 65 project-oriented coding challenges, each crafted to simulate real-world software development tasks. These challenges cover a range of frameworks, including React.js, Django, and Node.js, allowing for hands-on experience in building features and applications.

What sets this dataset apart is its focus on assessing problem-solving skills in complex, multi-file, long-context scenarios that mirror actual development environments. Unlike typical competitive programming datasets, HackerRank Astra does not provide public test cases, which prevents us from relying on hand-crafted test-time tactics. Evaluating performance with this dataset reveals whether reasoning abilities enhance success in algorithmic problem solving alone, or extend to more practical, industry-related coding tasks.

Figure 8 presents performance metrics such as pass@1 (the probability of successfully completing a task on the first attempt) and average scores (the mean proportion of test cases passed). The results illustrate the impact of chain-of-thought reasoning, with the o1-preview model achieving a 9.98% improvement in pass@1 and a 6.03-point gain in average score compared to GPT-4o. Further fine-tuning through reinforcement learning enhances o1’s performance, yielding a pass@1 of 63.92% and an average score of 75.80%—a 3.03% increase in pass@1 over o1-preview. These metrics demonstrate o1’s enhanced reasoning and adaptability, enabling it to address complex, industry-relevant software development tasks effectively.

### 5.2 SWE-Bench Verified

SWE-bench Verified is OpenAI’s preparedness team’s human-validated subset of SWE-bench that more reliably evaluates AI models’ ability to solve real-world software issues. This validated set of 500 tasks fixes certain issues with SWE-bench such as incorrect grading of correct solutions, under-specified problem statements, and overly specific unit tests. This helps ensure the benchmark accurately grades model capabilities.

To illustrate performance on this software task, we display the results presented in the o1 system card [ 4 ] as well as results from an early o3 checkpoint [ 13 ] . Because o1-preview was not trained to use code execution or file editing tools, the best-performing open-source scaffold at the time of initial implementation, Agentless was used. Unlike for IOI, no specialized test-time strategies was used for SWE-Bench verified. All models are given 5 tries to generate a candidate patch. If the model fails after 5 attempts, it is considered an incorrect attempt. All evaluations are averaged over 3 trials. We do not penalize the model for system failures (e.g., container hangs or grading failures), and we retry these rollouts until we can record a valid attempt.

As illustrated in Figure 9 , o1-preview demonstrates an 8.1% performance improvement on SWE-bench compared to gpt-4o, showcasing notable advancements in reasoning capabilities. With additional reinforcement learning compute applied during training, o1 achieves a further 8.6% improvement. Notably, o3, which was trained with significantly greater compute resources than o1, delivers an impressive 22.8% improvement over o1. These results underscore that enhanced reasoning skills extend beyond competitive programming challenges, proving their applicability to real-world tasks like software engineering.

## 6 Conclusion

Through the o-series large reasoning models, we demonstrate that chain-of-thought reasoning is a powerful strategy for improving performance in coding tasks, from competitive programming benchmarks such as CodeForces and IOI to complex software engineering challenges like SWE-bench and Astra. Our findings highlight that increasing reinforcement learning training compute, coupled with enhanced test-time compute, consistently boosts model performance to nearly match the best humans in the world. Given these results, we believe o-series large reasoning models will unlock many new use cases for AI in science, coding, math, and many other fields.

## Appendix A Authorship, credit attribution, and acknowledgments

##### Data Preparation:

Borys Minaiev, Ignasi Clavera, Lorenz Kuhn, Nat McAleese, Oleg Mürk, Szymon Sidor

##### IOI Model Training:

Ahmed El-Kishky, Mostafa Rohaninejad

##### Sampling Infrastructure:

Andre Saraiva, Hunter Lightman, Vineet Kosaraju, Wenda Zhou

##### Test-time Strategy:

Alexander Wei, Daniel Selsam, David Dohan, Francis Song, Ignasi Clavera, Max Schwarzer, Rhythm Garg, Rui Shu

##### Evaluation:

Andre Saraiva, Ignasi Clavera, Lorenz Kuhn, Nat McAleese

##### Leadership:

Jakub Pachocki, Jerry Tworek, Lukasz Kaiser, Mark Chen

##### o3 Model Development

o3 contributors [ 13 ] .

##### Acknowledgments:

We are grateful to the IOI committee for allowing us to enter our model, o1-ioi, in the 2024 International Olympiad in Informatics. We also extend our thanks to Wael Ewida, a member of the IOI technical committee, for hosting a portal that enabled us to submit our solutions under the same conditions as the contestants. Additionally, we appreciate the support of those who contributed to and maintained our sandboxed code execution, including Taylor Gordon, Oleg Boiko, John Rizzo, Paul Ashbourne, Leo Liu, Alexander Prokofiev, and Scottie Yan. We also extend our gratitude to Chris Orsinger and Michelle Fradin for their contributions to data efforts. Finally, we would like to express our sincere gratitude to everyone involved in the reinforcement learning reasoning efforts for o1 and o3, whose dedication and expertise were instrumental in advancing this work.

## Appendix B Additional CodeForces Details

In order to compare our models to human competitive programmers, we simulate contests. This section provides details of how the simulation is performed, how the overall score and ratings are calculated, as well as the per-contest results.

### B.1 Data

For our test set we use “Division 1” contests from late 2023 and 2024, all of which occurred after the o3 training set data cut-off. As a redundant additional check, we used embedding search to confirm that the test problems have not been seen by the model during training. We excluded one contest that contained an interactive problem for which grading was inconvenient, but otherwise included all post-cut-off Division 1 problems to which we had access at the time. During training we used a validation set of primarily Division 2 problems; when that set indicated that performance was very strong we built and evaluated the Division 1 set presented here.

### B.2 Grading

We run the complete set of tests for each problem, and have confirmed that our test environment closely matches the official CodeForces grading service, including by manually submitting solutions for the hardest problems to the official CodeForces graders.

Following AlphaCode [ 6 ] we allow the model to make 10 independent submissions against the full test set and mark a problem as solved if any one of those 10 passes. This is close to but not strictly the same as the human affordance, as human participants see only the results of the pre-tests during the competition. However in Division 1 contests the pre-tests are typically “strong” (highly correlated with full tests), and in our results the number of failures before a passing submission is typically small (see 1 ). We did not have access to labels for which test cases were pre-tests.

### B.3 Thinking Time

Competitors receive a higher score for submitting their solutions faster. Because models can think in parallel and simultaneously attempt all problems, they have an innate advantage over humans. We elected to reduce this advantage in our primary results by estimating o3’s score for each solved problem as the median of the scores of the human participants that solved that problem in the contest with the same number of failed attempts.

We could instead use the model’s real thinking time to compute ratings. o3 uses a learned scoring function for test-time ranking in addition to a chain of thought. This process is perfectly parallel and true model submission times therefore depend on the number of available GPU during the contest. On a very large cluster the time taken to pick the top-ranked solutions is (very slightly more than) the maximum over the thinking times for each candidate submission. Using this maximum parallelism assumption and the sequential o3 sampling speed would result in a higher estimated rating than presented here. We note that because sequential test-time compute has grown rapidly since the early language models, it was not guaranteed that models would solve problems quickly compared to humans, but in practice o3 does.

### B.4 Estimated Rating

The CodeForces rating system is described by the creator in three blog posts [ 8 , 9 , 10 ] . Ratings are similar to the Elo system and satisfy the property that if competitor A A has rating R A R_{A} and competitor B B has rating R B R_{B} then the probability that A A ranks better than B B any final contest standings is estimated as 1 10 R B − R A 400 \frac{1}{10^{\frac{R_{B}-R_{A}}{400}}}

To find the model rating we first calculate the rank of the model in each of the test contest from the total contest score (described above) and then directly maximize the likelihood of the observed rankings and human ratings with respect to the model rating using the equation above. We average to ensure that contests with more participants are not over-weighted.

We validated that this recovers known human ratings based on their contest performance and also gives similar values to linearly predicting participant rating from their average solve rate.

### B.5 Percentile performance

Codeforces maintains a global leaderboard of active participants, and an estimated rating can be used to compare to that group. We can also directly compare the solve rate of o3 in our test contests to the other participants in those contests. Figure 10 shows both these comparisons. Each point is a person that competed in at least 8 of the test contests. We show their average solve rate over contests that they entered against their rating, as well as the rating thresholds for key performance levels. The very best human competitors remain much stronger than o3, with solve rates in excess of 85%, but both ratings and solve rates indicate that o3 would rank among the top 200 active participants worldwide.

### B.6 Per Problem Breakdown

## Appendix C IOI Submissions

This section presents the solutions generated by o1-ioi during the 2024 International Olympiad in Informatics.

### C.1 Nile

All 100 possible points for Nile were scored in a single submission.

⬇

### C.2 Message

This solution achieved a score of 79.64 out of 100, with full marks awarded for subtask 1 and partial credit received for subtask 2.

⬇

### C.3 Tree

A total of 30 points were scored on Tree across two separate submissions.

#### C.3.1 Submission 1

The first submission achieved a score of 17 out of 100, with points earned from subtasks 1 and 4.

⬇

#### C.3.2 Submission 2

Submission 2 achieved 13 of 100 points on subtask 2. ⬇

### C.4 Hieroglyphs

A total of 44 points was scored on Hieroglyphs across two separate submissions.

#### C.4.1 Submission 1

In the first submission, a score of 34 out of 100 points was achieved, distributed across subtasks 1, 2, and 4.

⬇

#### C.4.2 Submission 2

In the second submission, the model scored 10 points on subtask 3.

⬇

### C.5 Mosaic

A total of 42 points were scored on Mosaic across two separate submissions.

#### C.5.1 Submission 1

The first submission achieved a score of 22 out of 100, with points distributed across subtasks 1, 2, and 4.

⬇

#### C.5.2 Submission 2

The model scored 20 points in the second submission on subtasks 1, 3, and 5. ⬇

### C.6 Sphinx

A total of 71.5 points were scored on Sphinx across two separate submissions.

#### C.6.1 Submission 1

The first submission achieved a score of 50 out of 100, with 50% partial credit earned on all subtasks.

⬇