LIMO: Less is More for Reasoning — part 2/2

Cleaned-text code points [15769, 31427). Bibliography omitted here; source.md includes it.

From our initial collection of 2125 questions, we selected the highest-scoring solution for each problem, ranked these pairs by quality score, and extracted the top 800 to form the LIMO Dataset. This curation process embodies our Less-Is-More philosophy—prioritizing demonstration quality over quantity to enhance complex reasoning capabilities.

## 4 Training Recipe

Based on the Less-Is-More principle, a model with substantial reasoning knowledge from pre-training and the ability to perform long-chain reasoning at test time can develop robust reasoning abilities. With exposure to just a few hundred carefully selected SFT examples, the model learns to integrate meta-reasoning tasks into cohesive reasoning chains.

We fine-tune Qwen2.5-32B-Instruct using supervised fine-tuning on our LIMO dataset. We set as our sequence length limit because all SFT response sequences remain under 16,384 tokens. The training process employs full-parameter fine-tuning with DeepSpeed ZeRO-3 optimization ( Rajbhandari et al., 2020 ) and FlashAttention-2 ( Dao, 2023 ) . For optimization, we utilize a learning rate of 5.0e-6 with a cosine decay schedule. We deliberately omit the warmup phase to facilitate rapid adaptation to the high-quality reasoning examples in our dataset. The model is trained for 15 epochs with a batch size of 64 examples to balance computational efficiency and stable convergence.

## 5 Evaluation Framework

We establish a comprehensive evaluation framework to assess our models’ mathematical reasoning capabilities across various dimensions. Our framework encompasses both in-domain and out-of-distribution evaluations, utilizing established benchmarks alongside novel multilingual tests to thoroughly examine generalization capabilities beyond the training distribution.

##### In-domain Evaluation

To comprehensively assess the models’ performance across various reasoning capabilities, we have established a diverse evaluation framework encompassing both traditional and novel benchmarks. Our primary evaluation suite includes several well-established mathematical competitions and benchmarks: the American Invitational Mathematics Examination (AIME24), MATH500 ( Hendrycks et al., 2021 ) , and the American Mathematics Competitions (AMC23).

##### Out-of-distribution Evaluation

To rigorously evaluate OOD performance, we select benchmarks differing from our training data across three categories. First, we include diverse mathematical competitions like OlympiadBench ( He et al., 2024 ) to test performance on different mathematical challenges. Second, to minimize data contamination, we construct novel multilingual benchmarks using recent examination problems: CHMath (2024 Chinese High School Mathematics League), Gaokao (China’s 2024 College Entrance Exam), Kaoyan (Chinese Graduate School Entrance Exams), and GradeSchool (our new elementary mathematics benchmark). These Chinese-language problems introduce an additional OOD dimension, assessing both cross-distribution and cross-lingual reasoning capabilities. Third, we incorporate multi-disciplinary benchmarks including Minerva ( Lewkowycz et al., 2022 ) (undergraduate-level STEM) and GPQA ( Rein et al., 2023 ) to evaluate the transfer of mathematical reasoning skills to broader contexts beyond our training domain.

##### Performance metrics

We evaluate performance using the pass@1 metric in a zero-shot chain-of-thought setting across all benchmarks. For larger benchmarks (MATH500, OlympiadBench, Gaokao, Kaoyan, GradeSchool, MinervaMath, and GPQA), we employ greedy decoding with a single sample. For smaller benchmarks (less than 50 problems: AIME24, AMC23, and CHMath), we generate 4 samples (temperature=0.6) and calculate the unbiased pass@1 metric ( Chen et al., 2021 ) . We use rule-based evaluations for numerical answers and an LLM-based evaluator for complex answer formats. All evaluations maintain a 32,768 token maximum output length.

## 6 Experiment

### 6.1 Baselines

We compare LIMO against a comprehensive set of baselines with several prominent models. These include OpenAI-o1-preview ( OpenAI, 2024 ) , a large language model that has demonstrated advanced mathematical reasoning abilities across various complex tasks; QwQ-32B-Preview ( Team, 2024b ) , a model specifically designed for mathematical problem-solving with strong reasoning capabilities; and Qwen2.5-32B-Instruct , which serves as our base model for comparative analysis.

To investigate the impact of training data efficiency, we conduct comparative experiments using mainstream open-source reasoning datasets for supervised fine-tuning on our base model. For a fair comparison, all experiments use the same LLM backbone as LIMO, ensuring that performance differences are solely attributable to the training data characteristics. These comparative datasets include OpenThoughts-114k ( Team, 2025a ) , a synthetic reasoning dataset containing 114k examples covering mathematics, science, coding, and puzzles, with solutions following a structured reasoning format generated by DeepSeek-R1; and NuminaMath-100k , a randomly selected 100k subset of NuminaMath-CoT, featuring mathematical problems ranging from Chinese high school exercises to international mathematics olympiad competitions, with each solution following a Chain of Thought (CoT) format ( Wei et al., 2022 ) .

These datasets contain substantially more samples than LIMO’s training set (800 examples), allowing us to examine the relationship between data quantity and model performance.

### 6.2 Main Results

Our experimental results demonstrate LIMO’s superior performance across both in-domain and out-of-domain tasks, as shown in Table 1 .

##### In-domain Performance

On in-domain tasks, LIMO achieves the best results across all benchmarks. For AIME24, LIMO achieves 63.3% accuracy, outperforming QwQ-32B-Preview (50.0%) and OpenAI-o1-preview (44.6%) by significant margins. On MATH500, LIMO achieves a notable 95.6% accuracy, surpassing QwQ-32B-Preview (89.8%) and OpenAI-o1-preview (85.5%). The performance gap is even more pronounced on AMC23, where LIMO reaches 96.3% accuracy compared to QwQ-32B-Preview’s 83.6%.

##### Out-of-domain Generalization

LIMO demonstrates strong generalization capabilities across diverse out-of-domain tasks. On OlympiadBench, LIMO achieves 67.6% accuracy, significantly outperforming QwQ-32B-Preview (58.5%) and the base model (45.3%). Similar improvements are observed on other challenging benchmarks such as CHMath (84.2% vs 68.5%) and GradeSchool (76.2% vs 63.8%). Notably, LIMO maintains competitive performance even on GPQA, where it achieves 70.7% accuracy, close to OpenAI-o1-preview’s leading score of 73.3%.

##### Comparison with Larger Datasets

Our experiments reveal that despite larger scale, both baseline datasets underperform compared to LIMO. NuminaMath-100k shows significant degradation (32.3% vs. base model’s 49.9%) due to uncurated reasoning chains, while OpenThoughts-114k achieves suboptimal results (58.3%) probably due to unfocused problem selection. In contrast, LIMO’s carefully curated 800 problems yield superior performance (78.1%), demonstrating that targeted selection are more crucial than data quantity for developing robust reasoning capabilities.

##### Overall Performance

LIMO achieves the highest average performance of 78.1% across all benchmarks, substantially outperforming OpenAI-o1-preview, QwQ-32B-Preview, and other baselines. This comprehensive evaluation demonstrates that LIMO’s carefully curated training approach with just 800 examples can outperform models trained on datasets that are orders of magnitude larger.

### 6.3 Analysis

#### 6.3.1 RQ1: Impact of Reasoning Chain Quality

To further validate and compare the effectiveness of our LIMO method, we first focus on the selection of high-quality reasoning chains. We investigate what characteristics define superior reasoning chains that lead to better model performance through a controlled comparative study of solutions with varying quality for identical problems.

For this analysis, we select 500 questions from the LIMO dataset that each have multiple correct solutions generated by diverse models, ensuring we have varied yet accurate approaches to the same problems. We collect and categorize these solutions into five distinct quality levels (L1-L5, with L5 being the highest) based on our rule-based scoring system described in Section 3.1.2 .

Results in Figure 3 show clear correlation between reasoning quality and model performance. L5-trained models achieve highest results on both AIME24 and MATH500, with performance decreasing consistently with each quality level. The substantial gap between L5 and L1 solutions demonstrates that reasoning chain quality significantly influences model performance, underscoring the importance of curating high-quality training data.

#### 6.3.2 RQ2: Impact of Question Quality

We hypothesize that more challenging problems foster complex reasoning chains and enhanced knowledge integration. To test this, we examine how question difficulty affects models’ reasoning capabilities. We select three sets of 500 problems with increasing difficulty: Simple-500 (MATH levels 1-2), Complex-500 (MATH levels 3-5), and Advanced-500 (AIME problems). We verify the difficulty gradient by evaluating various LLMs, observing declining accuracy and increasing solution length across these sets. We then use DeepSeek-R1 to generate high-quality solutions for each set and fine-tune Qwen2.5-32B-Instruct on them. Results in Figure 4 show that modifying problem selection alone leads to 16% accuracy improvement on AIME2024, reaching 51.5%. Notably, the model fine-tuned on Advanced-500 achieves 91.2% on MATH500 despite no in-domain training data, suggesting that reasoning improvements from increased problem difficulty generalize across datasets.

#### 6.3.3 RQ3: LLM Backbone - Pre-trained Knowledge

Building on our LIMO hypothesis that emphasizes latent prerequisite knowledge within model parameters, we examine how pre-training data affects a model’s ability to leverage minimal exemplars for math reasoning. To isolate pre-training impact while controlling for architecture and fine-tuning, we compare two 32B-parameter models: Qwen1.5-32B-Chat ( Team, 2024a ) and Qwen2.5-32B-Instruct (LIMO’s base model). While sharing identical architecture, Qwen2.5 features improved pre-training data quality, particularly in mathematical and code-related content. We SFT both models using identical LIMO datasets and evaluate on AIME2024 and MATH500 benchmarks.

Results in Figure 5 show pre-trained model choice dramatically impacts reasoning performance. LIMO (built on Qwen2.5) achieves 63.3% accuracy on AIME2024, a 54.1 percentage point improvement over Qwen1.5’s 9.2%. On MATH500, LIMO reaches 95.6% accuracy, surpassing Qwen1.5 by 30.4 percentage points. These substantial improvements confirm that Qwen2.5’s enhanced pre-training creates a stronger foundation for mathematical reasoning, aligning with our hypothesis that richer pre-trained knowledge enables more effective utilization of minimal exemplars.

It is hypothesized that increasing the number of parameters in a large language model enhances its capacity for deep reasoning and overall performance in complex tasks. To investigate this hypothesis, we fine-tune models from the Qwen2.5-Instruct series of varying sizes—3B, 7B, 14B, 32B, and 72B—using the LIMO dataset with 800 high-quality samples. The models are subjected to the same supervised fine-tuning recipe to ensure that any observed differences in performance are primarily attributable to model size.

#### 6.3.4 RQ4: LLM Backbone - Model Size

Figure 6 summarizes the performance of the models on both benchmarks. The performance on AIME24 shows a marked increase with model size, rising from 2.5 for the 3B model to 68.3 for the 72B model. This supports the hypothesis that larger models are better able to handle the deep reasoning required by competition-level math problems. While the improvements on MATH500 are also evident, the gains are less dramatic, suggesting that even smaller models can achieve high accuracy on easier benchmarks like MATH500. The difference in performance between the 32B and 72B models is marginal on MATH500 (95.6% vs. 94.8%) and slightly less pronounced on AIME24 (63.3% vs. 68.3%). This indicates a potential saturation point, where increasing the number of parameters further may yield diminishing returns in fine-tuning performance for these benchmarks.

#### 6.3.5 RQ5: Sample Efficiency

Our experiments reveal that a surprisingly small number (i.e. 800) of samples can elicit competition-level mathematical reasoning, though the lower bound for maintaining effective performance remains an open question. We explore the impact of dataset size on fine-tuning efficacy by systematically varying the number of training samples. To optimize performance with constrained dataset sizes, we rank all 2,125 questions in the LIMO-pool dataset based on their best reasoning chains’ quality scores. From this ranked set, we select top-ranked subsets of varying sizes—400, 800, 1,200, 1,600, and 2,000 questions—to construct five datasets (LIMO-400 through LIMO-2k). Each dataset is used to fine-tune our base model following an identical training recipe to ensure comparability.

Figure 7 and Figure 8 present the performance of models fine-tuned on different dataset sizes. Several key observations emerge: (1) Fine-tuning with just 400 samples yields dramatic improvement over the base model, increasing AIME24 accuracy from 16.5 to 57.5 and MATH500 accuracy from 79.4 to 94.8. (2) While performance continues to improve with larger datasets, we observe diminishing returns beyond 800 samples, with only marginal increases between LIMO-800 and LIMO-1k2 (+0.9 on AIME24, -0.2 on MATH500), suggesting an early plateau effect. (3) The highest dataset size (2k) provides the best overall performance (69.6 on AIME24, 95.8 on MATH500), but with minimal improvement over 1.6k, indicating that beyond a certain threshold, additional data contributes less significantly to fine-tuning gains. These findings highlight the efficiency of high-quality, carefully selected data in enhancing LLM mathematical reasoning. Future work could explore active learning strategies to further optimize sample efficiency.

## 7 Conclusion

Our work demonstrates that complex mathematical reasoning in LLMs can be achieved with surprisingly few examples, contradicting the prevailing assumption that massive training data is necessary. LIMO achieves competitive performance on challenging benchmarks using only 1% of the training data required by previous approaches. This confirms our Less-Is-More Reasoning Hypothesis: in knowledge-rich foundation models, sophisticated reasoning emerges through minimal but precisely orchestrated demonstrations that effectively utilize inference-time computation.

## 8 Acknowledgement

We would like to express our sincere gratitude to Yixiu Liu and Yiwei Qin for their valuable contributions to this research work. Their expertise, dedication, and collaborative spirit have significantly enhanced the quality of our study. Their insightful suggestions and technical assistance were instrumental in achieving our research objectives. We are also grateful to Run-Ze Fan for his diligent efforts in labeling our internal benchmark. We also wish to extend our appreciation to Haoyang Zou and Xuefeng Li for their valuable discussions during the early stages of this work. Their perspectives and insights helped shape the foundation of our research.