LIMO: Less is More for Reasoning — part 1/2
Cleaned-text code points [0, 15769). Bibliography omitted here; source.md includes it.
# LIMO: Less is More for Reasoning
###### Abstract
We challenge the prevailing assumption that complex reasoning in large language models (LLMs) necessitates massive training data. We demonstrate that sophisticated mathematical reasoning can emerge with only a few examples. Specifically, through simple supervised fine-tuning, our model, LIMO, achieves 63.3% accuracy on AIME24 and 95.6% on MATH500, surpassing previous fine-tuned models (6.5% on AIME24, 59.2% on MATH500) while using only 1% of the training data required by prior approaches. Furthermore, LIMO exhibits strong out-of-distribution generalization, achieving a 45.8% absolute improvement across diverse benchmarks, outperforming models trained on 100× more data. Synthesizing these findings, we propose the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning can emerge through minimal but strategically designed demonstrations of cognitive processes. This hypothesis suggests that the threshold for eliciting complex reasoning is not dictated by task complexity but rather by two key factors: (1) the completeness of the model’s pre-trained knowledge base and (2) the effectiveness of post-training examples in serving as “cognitive templates” that guide reasoning. 1 1 1 https://github.com/GAIR-NLP/LIMO
## 1 Introduction
Complex reasoning has long been considered one of the most challenging capabilities to instill in large language models (LLMs). While recent work has shown that LLMs can be effectively aligned with user preferences through relatively small amounts of instruction data ( Zhou et al., 2023 ) , teaching models to reason—particularly in mathematics and programming—is widely believed to require vastly more training examples ( Paster et al., 2023 ; Yue et al., 2024 ) . This conventional wisdom stems from the inherent complexity of reasoning tasks, which demand multi-step logical deduction, domain knowledge application, and structured solution paths. The resulting paradigm typically involves training on tens or hundreds of thousands of examples ( Yu et al., 2024 ; Li et al., 2024b ) , based on two fundamental assumptions: first, that mastering such complex cognitive processes requires extensive supervised demonstrations, and second, that supervised fine-tuning leads primarily to memorization rather than true generalization ( Zhang et al., 2024 ; Xu et al., 2024 ; Chu et al., 2025 ) .
While this approach has shown success, it imposes substantial computational costs. We argue this data-intensive paradigm may no longer be necessary. Recent advances have transformed how LLMs acquire and utilize reasoning knowledge, suggesting a more efficient approach. Two key developments have created conditions for reimagining reasoning in LLMs: (1) Knowledge foundation revolution : Modern foundation models now incorporate unprecedented amounts of mathematical content during pre-training ( Qwen et al., 2025 ; Yang et al., 2024 ; Wang et al., 2024 ) . Llama 2’s total training data was 1.8T tokens ( Touvron et al., 2023 ) , while Llama 3 used 3.7T tokens for mathematical reasoning ( Grattafiori et al., 2024 ) . Contemporary LLMs may already possess rich mathematical knowledge, transforming the challenge from knowledge acquisition to knowledge elicitation. (2) Inference-time computation scaling revolution : Techniques scaling longer reasoning chains reveal that effective reasoning requires substantial computational space during inference. Recent works ( OpenAI et al., 2024 ; Qin et al., 2024 ; Huang et al., 2024 ; Guo et al., 2025 ) show that extended reasoning chains significantly improve reasoning ability. Inference-time computation provides the crucial cognitive workspace where models can unpack and apply their pre-trained knowledge.
We hypothesize that successful reasoning emerges from the synergy of rich pre-trained knowledge and sufficient computational resources at inference time. These developments suggest that if models possess rich reasoning knowledge and adequate computational space, activating their reasoning capabilities may require only a small number of high-quality samples that encourage extended deliberation, rather than massive fine-tuning datasets. We propose the Less-Is-More Reasoning (LIMO) Hypothesis , identifying two critical factors determining the elicitation threshold for complex reasoning: (1) the latent presence of prerequisite knowledge within the model’s parameters, and (2) the effectiveness of minimal exemplars in demonstrating problem-solving processes that encourage extended deliberation. The sample efficiency of eliciting advanced reasoning is thus bounded by the model’s encoded knowledge foundation and its exposure to training samples that effectively utilize inference-time computation space.
Our LIMO approach begins with a rigorous data curation process designed to identify high-quality samples that maximize reasoning elicitation. Starting from a large pool of QA pairs, we implement a multi-layered filtering system: first conducting coarse difficulty filtering to eliminate trivial problems, then fine-grained difficulty assessment to identify challenging questions, followed by knowledge point diversification to ensure comprehensive coverage. Simultaneously, we filter reasoning chains based on logical coherence, step-by-step clarity, and solution accuracy. This meticulous process yields a compact yet potent dataset of just 800 training samples. With simple supervised fine-tuning (SFT) using only this curated dataset based on Qwen2.5-32B-Instruct ( Qwen et al., 2025 ) , LIMO achieves 63.3% accuracy on the highly challenging AIME benchmark and 95.6% on MATH, outperforming previous strong SFT-based models while using just 1% of their training data. These benefits generalize across diverse previously unseen scenarios, with LIMO consistently outperforming models trained on 100x more data. This demonstrates that complex reasoning abilities can be effectively elicited through minimal but carefully curated training samples.
The main contributions of this work are: (1) We establish the LIMO hypothesis, demonstrating that complex reasoning capabilities can be elicited with just hundreds of examples by leveraging rich mathematical knowledge in pre-trained models and detailed reasoning chains. (2) Following LIMO principles, we carefully construct the LIMO dataset and fine-tune Qwen2.5-32B-Instruct through simple SFT. Our experiments demonstrate that LIMO achieves highly-competitive performance on challenging mathematical reasoning benchmarks and maintains superior out-of-distribution performance. (3) Through extensive analysis and ablation studies, we validate the effectiveness of LIMO’s data selection principles and explore their applicability across different scenarios (varying foundation model knowledge, model sizes, and architectures). Additionally, we investigate LIMO’s minimum data requirements to achieve competitive performance. (4) We release our models, code, and curated datasets to support future research in data-efficient reasoning.
## 2 Related Work
### 2.1 Evolution of Mathematical Reasoning in LLMs
Large-scale training data has driven reasoning abilities in LLMs. During pretraining, reasoning is enhanced by relevant corpora ( Wang et al., 2024 ; Azerbayev et al., 2024 ; Paster et al., 2023 ; Shao et al., 2024 ) from textbooks, scientific papers, and mathematical code that capture diverse human cognitive patterns. Post-training research focuses on curating large-scale instruction data to teach reasoning ( Yue et al., 2023 ; Yue et al., 2024 ; Li et al., 2024a ; Yu et al., 2024 ) by scaling questions and solutions. While this approach has achieved significant gains, it has been criticized for relying on memorization rather than true generalization ( Mirzadeh et al., 2024 ; Zhang et al., 2024 ) . Mirzadeh et al. (2024) found that LLMs exhibit variance when responding to different instantiations of the same question, with performance declining when only numerical values are altered. This raises doubts about SFT methods’ generalization capability ( Chu et al., 2025 ) and whether LLMs can be true reasoners rather than merely knowledge retrievers ( Kambhampati, 2024 ) .
### 2.2 Test-time Scaling and Long Chain Reasoning
Instead of focusing on scaling model parameters and training data ( Kaplan et al., 2020 ) , recent work has shifted to exploring test-time scaling ( OpenAI, 2024 ; Snell et al., 2024 ) , i.e., increasing the number of tokens to improve performance. This can be achieved by augmenting LLMs with methods such as parallel sampling ( Brown et al., 2024 ; Wang et al., 2022 ; Li et al., 2022 ) or symbolic tree search ( Hao et al., 2023 ; Chen et al., 2024 ; Yao et al., 2023 ) to enhance reasoning ability. Furthermore, OpenAI (2024) ; Guo et al. (2025) explore training LLMs using reinforcement learning to generate long CoT, which often include self-reflection, verification, and backtracking—processes commonly employed by humans when solving complex problems. This approach not only innovates the training paradigm for LLMs but also provides a new form of training data to augment their reasoning ability. Our work demonstrates that this long CoT exhibits high-quality characteristics in eliciting the inherent reasoning abilities of LLMs.
### 2.3 Data Efficiency in Language Models
Zhou et al. (2023) demonstrates that with just 1,000 carefully curated prompts and responses, models can learn to follow specific formats and generalize well to unseen tasks. The findings emphasize the importance of quality over quantity in the alignment process. However, whether this lesson can be applied to reasoning tasks remains uncertain, given the potential high computational complexity of such tasks ( Merrill & Sabharwal, 2024 ; Xiang et al., 2025 ) . While some work on reasoning highlights the importance of quality during the curation of training data ( Zhou et al., 2024 ; Yu et al., 2024 ) , the quantity of such data is still much larger compared to that in LIMA. Our work extends the ideology of LIMA to reasoning tasks by investigating what constitutes high-quality questions and solutions, and demonstrates that the reasoning ability of LLMs can be enhanced in a highly data-efficient manner.
## 3 LIMO Dataset
We formalize the Less-Is-More Reasoning (LIMO) Hypothesis as follows: In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes. This hypothesis rests on two fundamental premises: (I) The latent presence of prerequisite knowledge within the model’s parameter space (II) The quality of reasoning chains that precisely decompose complex problems into detailed, logical steps, making the cognitive process explicit and traceable. To validate this hypothesis, we propose a systematic approach to construct a high-quality, minimal dataset that can effectively elicit the model’s inherent reasoning capabilities.
### 3.1 High-Quality Data Curation
In this paper, we focus on reasoning tasks with verifiable answers. Given a question q ∈ 𝒬 q\in\mathcal{Q} , we aim to generate an answer a ∈ 𝒜 a\in\mathcal{A} via a reasoning chain r ∈ ℛ r\in\mathcal{R} consisting of intermediate steps { s 1 , s 2 , … , s n } \{s_{1},s_{2},...,s_{n}\} . This reasoning process is formalized as f : 𝒬 → ℛ × 𝒜 f:\mathcal{Q}\rightarrow\mathcal{R}\times\mathcal{A} . Dataset quality depends on both question quality and solution quality. Our curation process constructs a deliberately small, high-quality dataset 𝒟 = { ( q i , r i , a i ) } i = 1 N \mathcal{D}=\{(q_{i},r_{i},a_{i})\}_{i=1}^{N} to validate our LIMO hypothesis that prioritizes quality over quantity.
#### 3.1.1 Question Selection
We hypothesize that high-quality questions q ∈ 𝒬 q\in\mathcal{Q} should naturally elicit extended reasoning processes. Our selection criteria prioritize challenging problems that foster complex reasoning chains and knowledge integration, while also considering problems that promote exploration of diverse problem-solving approaches.
To implement these criteria effectively, we first assembled a comprehensive pool of candidate problems from various established datasets: NuminaMath-CoT ( Li et al., 2024b ) , featuring meticulously annotated problems from high school to advanced competition levels; DeepScaleR ( Luo et al., 2025 ) , consists of approximately 40,000 unique mathematics problem-answer pairs; AIME historical examination problems before 2024, known for its extremely challenging and integrative problems spanning multiple mathematical domains; MATH ( Hendrycks et al., 2021 ) , encompassing various competitive mathematics problems from prestigious contests; and several questions including Chinese elementary, middle school, high school, and undergraduate-level exercises and examination papers.
From this extensive initial collection, we implemented a systematic multi-stage filtration pipeline (Figure 2 ). Starting with a corpus of tens of millions of mathematical problems, we first applied a baseline difficulty filter using a short-CoT mathematical model, Qwen2.5-Math-7B-Instruct ( Yang et al., 2024 ) . Problems that this model solved correctly within four attempts were excluded, ensuring that only non-trivial problems remained. Next, we subjected the filtered problems to a more rigorous evaluation using a stronger reasoning model, DeepSeek-R1-Distill-Qwen-32B ( Guo et al., 2025 ) . For each remaining problem, we sampled 32 solution attempts and used the empirical success rate as a difficulty indicator. Problems that were successfully solved in only 1–3 out of 32 attempts were retained, yielding a refined dataset of 2,125 problems, which composed our initial data pool (referred to as LIMO-Pool). To ensure dataset integrity, we conducted thorough deduplication against all evaluation benchmarks with n-gram matching, confirming no overlap existed.
#### 3.1.2 Reasoning Chain Construction
The quality of reasoning chains fundamentally impacts the effectiveness of large language model training. To develop our solution dataset, we employed three state-of-the-art reasoning models—DeepSeek R1, DeepSeek-R1-Distill-Qwen-32B ( Guo et al., 2025 ) , and QwQ-32B ( Team, 2025b ) —sampling multiple solutions from each to generate diverse reasoning approaches. Subsequently, all the authors conducted a comprehensive analysis of these filtered solutions through collaborative examination. Through careful observation and systematic review, we identified several key characteristics that distinguish high-quality reasoning chains:
Elaborated Reasoning : Comprehensive exploration of logical steps without premature conclusions
Self-Verification : Regular validation of intermediate results and logical consistency
Exploratory Approach : Consideration of multiple possibilities before reaching conclusions
Adaptive Granularity : Appropriate detail level across simple and complex deductions
To quantify these qualities, we implemented a rule-based scoring system that calculated weighted metrics for each dimension. Elaborated Reasoning was measured by solution length (30% weight); Self-Verification through frequency of validation-related words like ”check” and ”verify” (20% weight); Exploratory Approach by counting tentative expressions such as ”perhaps” and ”might” (25% weight); and Adaptive Granularity via connective phrases like ”therefore” and ”since” (25% weight). All keyword frequencies were normalized by text length to ensure fair comparison across solutions of different sizes.