Competitive Programming with Large Reasoning Models — part 1/3

Cleaned-text code points [0, 15412). Bibliography omitted here; source.md includes it.

# Competitive Programming with Large Reasoning Models

###### Abstract

We show that reinforcement learning applied to large language models (LLMs) significantly boosts performance on complex coding and reasoning tasks. Additionally, we compare two general-purpose reasoning models — OpenAI o1 and an early checkpoint of o3 — with a domain-specific system, o1-ioi, which uses hand-engineered inference strategies designed for competing in the 2024 International Olympiad in Informatics (IOI). We competed live at IOI 2024 with o1-ioi and, using hand-crafted test-time strategies, placed in the 49th percentile. Under relaxed competition constraints, o1-ioi achieved a gold medal. However, when evaluating later models such as o3, we find that o3 achieves gold without hand-crafted domain-specific strategies or relaxed constraints. Our findings show that although specialized pipelines such as o1-ioi yield solid improvements, the scaled-up, general-purpose o3 model surpasses those results without relying on hand-crafted inference heuristics. Notably, o3 achieves a gold medal at the 2024 IOI and obtains a CodeForces rating on par with elite human competitors. Overall, these results indicate that scaling general-purpose reinforcement learning, rather than relying on domain-specific techniques, offers a robust path toward state-of-the-art AI in reasoning domains, such as competitive programming.

## 1 Introduction

Competitive programming is widely recognized as a challenging benchmark for evaluating reasoning and coding proficiency [ 2 ] . Solving complex algorithmic problems demands advanced computational thinking and problem solving skills. Moreover, these problems are also objectively gradable, making it an ideal testbed to assess the reasoning capabilities of AI systems.

Recent work on program synthesis with large language models [ 1 ] has demonstrated that even relatively general models, ranging from 244M to 137B parameters, can generate short Python scripts from natural language instructions. Importantly, performance improves log-linearly with model size, and fine-tuning significantly boosts accuracy. Concurrently, Codex [ 2 ] , an early code-focused LLM, excelled at Python program generation and powered GitHub Copilot. Further progress came from AlphaCode [ 7 ] , which tackled competitive programming tasks using large-scale code generation and heuristics at inference, and the subsequent AlphaCode2 [ 6 ] , whose improvements nearly doubled AlphaCode’s solved problems and placed it in the 85th percentile on the CodeForces platform. Both AlphaCode systems used large-scale sampling of up to a million candidate solutions per problem before selecting their top 10 submissions with a hand-engineered test-time strategy.

Since then, significant progress has been made in harnessing reinforcement learning to improve LLMs’ reasoning skills. This has led to the emergence of large reasoning models (LRMs): language models trained via reinforcement learning to “reason” and “think through” extended chains of thought. In particular, OpenAI’s o1 [ 4 , 12 ] and its soon-to-be-released successor o3 [ 13 ] use chain-of-thought reasoning to tackle intricate tasks such as mathematics and coding. Work by DeepSeek-R1 [ 3 ] and Kimi k1.5 [ 15 ] independently illustrates how learning chain-of-thought boosts performance on both mathematical and programming challenges.

An open question is how domain-specific, hand-engineered inference strategies compare to learned approaches that models generate and execute on their own. We have three systems available that can shed light on this question: o1, o1-ioi, and early checkpoints of o3. OpenAI o1 was the first large reasoning model and used general purpose methods to improve programming performance. Building on this foundation, o1-ioi was a fine-tuned system tailored to compete in the 2024 International Olympiad in Informatics (IOI) and used test-time strategies similar to those used in the AlphaCode system. This specialization led to strong performance improvements on both the 2024 IOI and competitive programming platforms such as CodeForces . Subsequent advances led to the development of o3, which has significantly advanced the reasoning capabilities of AI models. Unlike o1-ioi or AlphaCode, o3 does not depend on coding-specific test-time strategies defined by humans. Instead, we found that complex test-time reasoning strategies emerged naturally from end-to-end RL, leading to unprecedented performance on competitive programming benchmarks.

This report provides a high-level overview of the importance of reasoning in coding tasks such as competitive programming, the progress of OpenAI’s large reasoning models in programming ability, and our evaluation methodology and results on various competitive programming and coding benchmarks.

## 2 OpenAI o1

We start with OpenAI o1, a large language model trained with reinforcement learning to tackle complex reasoning tasks. By generating an extended internal chain of thought before answering [ 16 ] , o1 resembles a human who methodically works through a challenging problem step by step. Reinforcement learning refines this chain-of-thought process, helping the model identify and correct errors, break down complex tasks into manageable parts, and explore alternate solution paths when an approach fails. These in-context reasoning capabilities substantially boost o1’s overall performance on a wide range of tasks.

Additionally, OpenAI o1 is trained to use external tools [ 14 ] , especially for writing and executing code in a secure environment. 1 1 1 https://platform.openai.com/docs/assistants/tools/code-interpreter This capability lets o1 verify whether its generated code compiles, passes provided test cases, and meets other correctness checks. By testing and refining its outputs, o1 iteratively improves its solutions over the course of a single sample.

### 2.1 CodeForces Benchmark

CodeForces is a programming competition website that hosts live contests. It is internationally competitive and frequented by some of the best competitive programmers in the world.

To assess our models’ competitive programming abilities, we simulated CodeForces contests under conditions that closely mirrored real competitions. This included using the full test suite for each problem and enforcing appropriate time and memory constraints for solutions.

Our evaluation focused on Division 1 contests from 2024 and December 2023, ensuring all test contests occurred after the data cut-off for both pretraining and RL. Additionally, we conducted a contamination check as a sanity measure, leveraging the OpenAI embedding API to verify that test problems had not been seen during training.

We compared o1 against a non-reasoning LLM (gpt-4o) and an earlier reasoning model (o1-preview). Figure 1 shows how both o1-preview and o1 dramatically outperform gpt-4o, highlighting the effectiveness of reinforcement learning for complex reasoning. The o1-preview model achieved a CodeForces rating of 1258 (62nd percentile) — up from gpt-4o’s 808 (11th percentile). Further training pushed o1’s rating to 1673 (89th percentile), establishing a new milestone for AI performance in competitive programming.

In Appendix B we provide additional details of which problems our models can solve and how ratings were calculated.

## 3 OpenAI o1-ioi

During our development and evaluation of OpenAI o1, we found that increasing both the amount of reinforcement learning (RL) compute and test-time inference compute consistently improved model performance.

As shown in Figure 2 , scaling RL training and extending test-time inference led to marked gains, highlighting the importance of optimizing these two compute dimensions to push performance beyond conventional LLM pretraining.

Building on these insights, we created the o1-ioi system for competing at the 2024 International Olympiad in Informatics (IOI). In addition to continued RL training targeted at coding tasks, o1-ioi incorporates specialized test-time inference strategies engineered for competitive programming.

### 3.1 Coding RL Fine-tuning

Our first step extended the reinforcement learning phase of OpenAI o1, focusing on coding tasks. By dedicating additional training compute to programming problems, we bolstered the model’s ability to plan, implement, and debug more involved solutions. Concretely:

1. We resumed RL training from the OpenAI o1 checkpoint.

2. We specifically emphasized challenging programming problems, helping the model improve C++ generation and runtime checks.

3. We guided the model to produce outputs in the IOI submission format.

This added focus on coding allowed o1-ioi to write and execute C++ programs during inference. The model improved its reasoning by iteratively running and refining solutions, thereby strengthening both its coding and problem-solving skills.

### 3.2 o1-ioi Test-time Strategy

At a high level, we divided each IOI problem into its constituent subtasks, sampled 10,000 solutions from o1-ioi for each subtask, and then employed a clustering- and reranking-based approach to decide which solutions from this set to submit.

##### Problem formulation

For o1-ioi we chose to attempt to solve the individual subtasks of each problem separately, as the scoring for IOI is done on a subtask-by-subtask basis and gives each competitor the maximum score over all of their attempts on each subtask. To do this, we divided each IOI problem into its composite subtasks (using the divisions laid out in the scoring guide for each problem). This was done simply by creating one version of the document for each subtask with the information about the other subtasks removed.

##### Clustering

We clustered the generated solutions based on their outputs on model-generated test inputs. For each subtask, we first prompted the model to write random test input generators in C++ given the problem specification and subtask. We used these generators to generate 256 random test inputs. To ensure the validity of these test inputs, we then prompted the model to write test input validators in C++ that check, given a test input, whether it satisfies the subtask constraints. Finally, we accepted each test input that passes at least 75% of the validators. For each subtask, we generated 256 of these random test case inputs, and then clustered based on their outputs for these test cases. Any programs that matched each other’s outputs on all test inputs would be placed in the same cluster.

##### Reranking

We then implemented the reranking core of our test-time compute strategy. We scored each solution based on: • The quality of the solution according to a learned scoring function.

• Errors on model-generated test inputs.

• Failing the provided public test cases.

Each cluster was given a score defined as the average score of the samples it contained minus a penalty for each time a sample submission was attempted from that cluster. The weights of all of these penalties were tuned by random search on solutions to previous years’ IOI problems, by directly simulating the submission process.

##### Submission

We then submitted up to 50 (the maximum number allowed for human competitors) of these solutions in a round-robin fashion over subtasks, starting from the hardest. We selected the top-ranked solution in the top-ranked cluster for each given subtask. When a subtask was solved (meaning that the maximum score was attained), we ceased sampling on that subtask. When submitting solutions to any subtask that was a strict superset of a solved subtask, we would filter out any solutions that did not match the outputs on test inputs of the solved constituent subtasks, allowing us to rapidly narrow down candidate solutions on harder subtasks by rejecting those that would almost certainly have failed easier subtasks.

### 3.3 CodeForces Benchmark

Once again, we simulated CodeForces contests to evaluate o1-ioi’s coding abilities, closely mirroring contest conditions with the complete test suite for each problem and appropriate time and memory restrictions for solutions.

Figure 3 shows that o1-ioi reached a CodeForces rating of 1807, outperforming 93% of competitors — demonstrating clear improvements from additional RL training on coding tasks. When we applied a simple filter rejecting any solution that failed public tests, the rating rose to 2092 (96th percentile). Our complete test-time strategy pushed performance even further, attaining a rating of 2214 (98th percentile). These results confirm that domain-specific RL fine-tuning paired with advanced selection heuristics can significantly boost competitive programming outcomes.

### 3.4 IOI 2024 Live Competition

The o1-ioi system participated in the 2024 International Olympiad in Informatics (IOI) under the same conditions as human contestants. It had ten hours to solve six challenging algorithmic problems and was allowed up to 50 submissions per problem. We show the results in Figure 4 .

During the competition, our system generated 10,000 candidate solutions for each problem, and selected 50 submissions using our test-time selection strategy. This strategy prioritized submissions based on their performance on the IOI public test cases, model-generated test cases, and a learned scoring function. The model scored 213 points, placing it in the 49th percentile of the competition.

In comparison, selecting 50 random submissions would have yielded an average score of only 156 points, indicating that the selection strategy contributed nearly 60 additional points under the competition’s constraints.

When the submission limit was relaxed to 10,000 per problem, the model’s performance improved dramatically. Without employing any test-time selection strategy, it achieved a score of 362.14, surpassing the gold medal threshold and demonstrating the model’s potential. We show samples that yielded the 362.14 score in Appendix C .

## 4 OpenAI o3

Building on the insights gained from o1 and o1-ioi, we explore the limits of reinforcement learning (RL) training alone, without relying on human-engineered test-time strategies. While o1-ioi achieved strong results by combining additional RL fine-tuning with carefully designed test-time inference pipelines, its success hinged on human intervention to define and implement these strategies. We sought to explore the performance of a model even further trained with RL with the ability to autonomously develop and execute its own test-time reasoning strategies. To this end, we obtained access to early checkpoints of o3 [ 13 ] to evaluate on competitive programming tasks.

### 4.1 CodeForces Benchmark

We evaluate an early checkpoint of the o3 model on our CodeForces benchmark set, where each prompt includes the problem description, constraints, and any available sample test cases.

As shown in Figure 5 , further RL training provided a significant improvement over both o1 and the full o1-ioi system. Notably, the transition from the o1-ioi model to o3 resulted in a rating increase from 2214 (98th percentile) to 2724 (99.8th percentile), reflecting a substantial leap in competitive programming performance. This improvement demonstrates o3’s ability to solve a wider range of complex algorithmic problems with higher reliability, pushing its capabilities closer to top-tier human competitors on CodeForces .