s1: Simple test-time scaling — part 3/3

Cleaned-text code points [31457, 44075). Bibliography omitted here; source.md includes it.

We thank Ryan Marten for generating traces from DeepSeek r1 for s1.1 using Bespoke Curator ( Marten et al., 2025 ) . This work was partly conducted using the Stanford Marlowe GPU cluster ( Kapfer et al., 2025 ) , made possible by financial support from Stanford University. We thank Alexander M. Rush, Andrew Ilyas, Banghua Zhu, Chenglei Si, Chunting Zhou, John Yang, Ludwig Schmidt, Samy Jelassi, Suhas Kotha, Tengyu Ma, Xuechen Li, Yu Sun, and Yue Zhang for very constructive discussions.

## Appendix A s1.1

Seven days after our release of s1, we released s1.1. We regenerated traces for our 1,000 samples in s1K using DeepSeek r1 ( DeepSeek-AI et al., 2025 ) to create s1K-1.1 . We use the same training procedure to train our model s1.1 . Other updates since our launch include the release of o3 ( OpenAI, 2025 ) , LIMO ( Ye et al., 2025a ) , and AIME 2025. We consider all these new developments in Table 5 . We find that s1.1 performs significantly better than s1. We also tried distilling from Claude 3.7, which led to worse performance than from r1 (not reported).

## Appendix B Evaluation determinism

We run our evaluations using vLLM ( Kwon et al., 2023 ) as it is faster than the alternatives we tried. However, we find that even when using the same random seeds and greedy sampling, evaluation scores can change significantly across runs: • Different batch sizes causing different results see https://github.com/vllm-project/vllm/issues/5898

• Continuing generations causing different results see https://github.com/vllm-project/vllm/issues/11783

• Changes in tensor parallelism causing different results

As our model generates long reasoning traces prior to its answer, small numeric changes can snowball into large differences. We encounter many generations that are exactly the same for thousands of tokens and then suddenly differ in one token eventually ending up with an entirely different answer. To partly counter this issue we generally run our final evaluations using full precision unless otherwise indicated.

## Appendix C s1K details

### C.1 s1K summary

### C.2 Dataset composition for full 59K questions

### C.3 s1K grading prompt

To grade whether an example is correct for our dataset selection in §2 , we use the prompt in Figure 8 . We grade using Claude 3.5 except for the correctness among the final 1,000 samples, which we graded with Claude 3.7.

### C.4 s1K diversity selection

Algorithm 1 provides our algorithm for selecting data in our diversity selection stage. As mentioned in §2 , we also include samples from some specific benchmarks we perceive as high-quality. None of the samples overlap with our final evaluation.

### C.5 Decontamination

We filter all samples by checking for an 8-gram overlap between the selected examples and the evaluation benchmarks: MATH500, GPTQA Diamond, and AIME24. We exclude questions with more than an 8-gram overlap.

## Appendix D Training details

We take a model that has already been pretrained and instruction tuned and further finetune it for reasoning. Specifically, we use Qwen2.5-32B-Instruct ( Qwen et al., 2024 ) , which on math tasks generally matches or outperforms the larger Qwen2.5-72B-Instruct ( Qwen et al., 2024 ) or other open models ( Dubey et al., 2024 ; Groeneveld et al., 2024 ; Muennighoff et al., 2024 ) . We use token delimiters to separate the thinking stage from the answering stage. We enclose the thinking stage with <|im_start|>think and <|im_start|>answer ; both preceded and followed by a newline. Samples from our dataset are in §D.2 . We use basic fine-tuning hyperparameters: we train for 5 epochs with a batch size of 16 for a total of 315 gradient steps. We train in bfloat16 precision with a learning rate of 1 e − 5 1e-5 warmed up linearly for 5% (16 steps) and then decayed to 0 over the rest of training (299 steps) following a cosine schedule. We use the AdamW optimizer ( Loshchilov & Hutter, 2019 ) with β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 and weight decay of 1 e − 4 1e-4 . We do not compute loss on questions, only on reasoning traces and solutions. We ensure the sequence length is large enough to avoid cutting off any samples; a setting we ablate in §D.1 . The training takes just 26 minutes on 16 NVIDIA H100 GPUs.

### D.1 Training Ablations: Sequence length

Besides our scaling ablations in §5.2 , the main training hyperparameter we ablate is the sequence length used during training. We find that a shorter training sequence length leads to longer reasoning traces at test time. This is because when training with a shorter sequence length the answer section of the training sample is more commonly cut off. Inversely, when the training sequence length is longer, more samples appear in their entirety with the section where the model answers. Thus the model receives more gradient updates where it learns to generate an answer following its chain. This in turn leads to a higher log probability of the answer section at any point during the generation and thus shorter reasoning traces at test time. Performance-wise, we also find that the model trained with a longer sequence length performs better. Thus we opt for the longest training sequence length as it leads to better performance and makes inference more efficient by leading to shorter reasoning traces.

### D.2 Training Samples

, Table 10 , Table 11 contain training samples from s1K .

## Appendix E Test-time scaling details

### E.1 Sequential scaling ablations

#### Token-conditional control

One general approach is to simply tell a model in the prompt precisely how many tokens it should generate. Ideally, the model can keep track of its token count and adjust its generation to finish within the desired limits. We experiment with this approach by training a model with token instructions using the format in Figure 10 (left). We bucket the lengths of the reasoning traces from our 1,000 training examples into powers of two (rounded upwards) and add a corresponding instruction to the user prompt. For example, if the instruction says “Think for up to 2048 tokens”, then the reasoning trace has anywhere between 1024 and 2048 tokens. In Table 12 , we show that after training the model hardly follows the token instruction. It does sometimes generate more tokens when given a higher limit but often overshoots the limit. This may not be unique to our model as prior work suggests that OpenAI o1-mini can also not follow token instructions ( Zhang & Chen, 2024 ) . To prevent exceeding the limit, we test budget forcing the thinking to end once the limit is reached. This leads to perfect control ( Table 12 (lower)). With budget forcing, the scaling trend is also clearer as the model can no longer overshoot the limit when given a small thinking budget. This leads to better test-time scaling values for Token Prompting + budget forcing in Table 3 . To compute Control reported in Table 3 for token-conditional control variants we divide the number of times the thinking tokens in Table 12 are less than the upper limit by the total evaluations (2/5 for without intervention; 5/5 for with intervention).

#### Step-conditional control

Token instructions fail as current models cannot count tokens. To accommodate this lack of capability, we experiment with making the counting more coarse-grained. We partition the reasoning traces into steps and ask the model to think for a specific number of steps rather than tokens. We split our reasoning traces on double newlines into steps, which we find act as intuitive separators based on manual inspection of samples. We bucket our training samples into powers of 2 depending on their number of steps and add a corresponding step instruction following the format in Figure 10 (right). This format is based on early experiments, where we found the model to be more likely to adhere to the step limit when counting down (“3 steps left…2 steps left”) rather than counting up (“Step2…Step3…”). This is likely because if counting down, the final step is always 1, which will act as a strong prior to the model to finish its generation. If counting up, the final step before the answer varies, thus if the model does not remember the original step instruction, it may fail to stop. We conclude the following from our results in Table 13 : (1) The model still struggles to adhere to the step limit. The model sometimes simply continues counting into negative steps, e.g. “-1 steps left”. To solve this issue, we automatically stop the thinking process once 0 steps are reached and then force the model to transition to answering mode by appending the answer token delimiter ( §3 ). This leads to perfect step adherence (lower half of Table 13 ), yet problems remain. (2) The model compensates for fewer steps by making each step longer. For example, when forced to use up to 16 steps vs 256 steps, the model generates an average of 96 tokens per step vs 56. Despite this issue, more steps still clearly correlate with more total thinking tokens in Table 13 and better performance leading to a positive slope (3) Step instructions are more costly than other methods. The step delimiters require around 6 tokens each which for e.g. 64 steps adds up to a total of around 380 tokens. When ignoring the step delimiters in token counts as in Table 13 , the model still requires 7551 thinking tokens on average to achieve only 33.3% on AIME24. To compute Control reported in Table 3 for step-conditional control variants, we first decide that 100 tokens are an upper limit per step and then multiply this number by the steps instructed to arrive at a proxy total token limit, e.g. 1600 for 16 steps instructed. We then check whether the thinking tokens in Table 13 fit within the respective limit for each evaluation run (3/5 for without intervention; 5/5 for with intervention). For the model in Figure 7 , we use a model with step-conditional control trained on an earlier version of our data and using an earlier version of our evaluation codebase.

#### Class-conditional control

OpenAI exposes test-time compute control to users via a ‘‘reasoning_effort’’ API parameter with three possible settings: low, medium, and high. 3 3 3 https://github.com/openai/openai-python/blob/44d6210f101abedeb2dd68507fcffcb329df70ea/src/openai/types/chat/completion_create_params.py#L172 The OpenAI documentation also states that “Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.” suggesting that they are unable to control test-time compute with guarantees. Thus, maybe OpenAI simply adjusts the prompt or system instruction depending on the reasoning effort desired. In Table 14 , we show that separate prompts for short and long thinking allow us to control thinking time to some extent: Prompting the model to think for longer leads to longer thinking. However, it does not reliably improve performance and control is not precise. The current adherence to control may suffice when we only have three classes, but it might not scale to finer-grained classes. To compute Control reported in Table 3 for this method, we assume that prompting the model to think for a short time in Table 14 should produce fewer tokens than the default for AIME24, while the long prompt should produce more. As 8033 > 6109 8033>6109 and 9651 > 6109 9651>6109 , one out of two follows our expected control thus Control is 50%.

### E.2 Examples for rejection sampling ablation

## Appendix F Version Control

V2 → V3 (2025-03): • Added §A

• Added number of correct samples in §2

V1 → V2 (2025-02): • Added citations and other small writing changes

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .