Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process — part 3/6
Cleaned-text code points [31282, 46416). Bibliography omitted here; source.md includes it.
Unlike standard linear probing, to account for the input change, we introduce a small trainable rank-8 (linear) update on the input embedding layer. We freeze the pretrained language model and finetune both the linear classifier and the rank-8 update for the desired property. We refer to this as V(ariable)-probing and provide details in Appendix B . An illustration of the nece ( A ) \texttt{nece}(A) probing task is shown in Figure 6 .
We compute the V-probing accuracies on a language model pretrained from iGSM and compare them with the V-probing accuracies on a randomly-initialized transformer model. If the former accuracies are significantly higher, we conclude that the probing signals must have (or be very close to having) come from the pretrained weights, rather than the (lightweight) finetuning stage.
### 4.2 Probing Results and Findings
We present our probing results in Figure 7 . The probing accuracies are high for all the tasks, compared to majority guess and random-model probing — except for the very hard OOD cases (i.e., for large op where the model’s generation accuracies fall down to 80% anyways in Figure 3 ),
Result 4: model solves math problems like humans. We make the following observations: • When generating solutions, the model not only remembers which parameters have been computed and which have not ( value , known \texttt{value},\texttt{known} ) but also knows which parameters can be computed next ( can_next , nece_next \texttt{can\_next},\texttt{nece\_next} ). These abilities ensure that the model can solve the given math problem step by step, similar to human problem-solving skills.
• By the end of the problem description, the model already knows the full list of necessary parameters ( nece ). This indicates that the model has learned to plan ahead , identifying necessary parameters before starting to generate the solution. This aligns with human behavior, except that the model plans mentally while humans typically write this down. This further confirms that the model reaches the “level-1” reasoning skill discussed in Section 3 .
###### Remark 4.1 .
The mental process described can be compared to (out-of-context) knowledge manipulation [ 2 ] , which involves retrieving factual knowledge and performing single-step computations (e.g., retrieving two people’s birth dates to determine who was born earlier). Allen-Zhu and Li [2] found that even single-step computations cannot be performed mentally without a substantial number of pretrain samples. In contrast, this paper studies in-context reasoning and demonstrates that the model can execute very complex mental calculations.
Result 5: model learns beyond human reasoning skills. Remarkably, the model learns dep ( A , B ) \texttt{dep}(A,B) and can_next ( A ) \texttt{can\_next}(A) , even for parameters A A not necessary for answering the question, as shown in Figure 7(b) . This differs from human problem-solving, where we typically use backward reasoning from the question to identify necessary parameters, often overlooking unnecessary ones [ 19 ] . In contrast, language models can pre-compute the all-pair dependency graph dep ( A , B ) \texttt{dep}(A,B) mentally even before a question is asked. We consider this a “level-2” reasoning skill that is very different from human behavior or mental processes.
Thus, although this skill is not needed for solving the math problems and although no pretrain data teaches the model to compute “all-pair dependency” — fitting the data only requires computing necessary parameters — the model still discovers it after training. This enables the model to sort relationships among the things it hears, a skill that can be useful for future tasks (via instruction fine-tuning). To our knowledge, this may be the first evidence of a language model acquiring skills beyond those needed for learning its pretrain data; and this may be a preliminary signal of where the G in AGI can come from (generalizing to skills not taught in the pretrain data).
Corollary: the backward thinking process. A key question for AGI success is whether the “backward thinking process” (e.g., “because I want to compute X, but X depends on Y and Y depends on Z, so let me compute Z first”) needs to be explicitly included in the training data. This differs from CoT, where CoT breaks down complex computations into simpler steps, but planning is still required to decide which step to compute first.
Our findings suggest that, at least for grade-school math problems, with abundant data, this backward thinking process can be autonomously learned through language modeling, without needing to be directly included in the training data.
## 5 Result 6: Explain Model’s Mistakes
We further examine the relationship between our probing results and the model’s generated solutions, focusing on two questions: (1) When does the model answer correctly but include unnecessary parameters? (2) What causes incorrect answers? We aim to determine if such erroneous behavior of the model aligns with errors in the model’s mental process (via probing).
For the first question, given the model rarely produces solutions longer than necessary (see Figure 4 ), we turned to out-of-distribution reask data for evaluation. 17 17 17 Recall this re-samples a query after generating the problem, leading to a different set of necessary parameters. On this data, pretrained models produce an average of ∼ 0.5 \sim 0.5 unnecessary parameters per solution even for op = 32 \textsf{op}=32 (see Figure 4 ). We examined if these unnecessary parameters A A were incorrectly predicted as nece ( A ) = true \texttt{nece}(A)=\mathrm{true} in the probing task. Figure 8(a) reveals that this is often indeed the case, thus language models produce solutions with unnecessary steps due to errors in their mental planning phase .
For the second question, we focused on the model’s wrong solutions and their first wrong parameters . (Using synthetic data, we can easily identify such parameters.) Our findings in Figure 8(b) show that the model’s errors mainly stem from incorrectly predicting nece_next ( A ) \texttt{nece\_next}(A) or can_next ( A ) \texttt{can\_next}(A) as true in its internal states when such A A ’s are not ready for computation. 18 18 18 In Figure 8(b) , we focus on these “first wrong parameters” with correct label being can_next ( A ) = false \texttt{can\_next}(A)=\mathrm{false} or nece_next ( A ) = false \texttt{nece\_next}(A)=\mathrm{false} and present the probability that their probing also correctly predicts false \mathrm{false} . Low accuracy indicates that the model “thought” these parameters were ready for computation, but they were not. Result 6 ( Figure 8 ) . Combining these, we conclude: • Many reasoning mistakes made by the language model are systematic, stemming from errors in its mental process , not merely random from the generation process. • Some of the model’s mistakes can be discovered by probing its inner states even before the model opens its mouth (i.e., before it says the first solution step).
We also observe that GPT-4/4o makes similar mistakes by outputting unnecessary parameters or insisting on computing parameters A A with can_next ( A ) = false \texttt{can\_next}(A)=\mathrm{false} (see Appendix G ). This further hints that our findings may be applicable more broadly.
## 6 Result 7-8: Depth vs. Reasoning Length
Our controlled dataset enables a systematic exploration of the relationship between a language model’s depth and its reasoning length.
Recent studies have demonstrated that for knowledge storage and extraction, only model size matters (even for 2-layer transformers) [ 4 ] . Furthermore, both the seminal scaling-law paper by OpenAI [ 14 ] and theoretical studies in deep learning [ 5 ] suggest that model depth/width might have a minimal impact universally. Contrary to these findings, we present evidence that 19 19 19 Math reasoning data only occupies a tiny fraction of pretraining data for language models, thus one might not observe a difference if we only look at the perplexity as in the original scaling law paper [ 14 ] .
Result 7 ( Figure 9 ) . Language model depth is crucial for mathematical reasoning.
Specifically, we experimented with models of depths 4/8/12/16/20 and two sizes (a smaller size 1 and a larger size 2). 20 20 20 GPT2- ℓ \ell - h h represents an ℓ \ell -layer, h h -head, 64 h 64h -dimensional GPT2 model. Size-1 models are GPT2-4-21, GPT2-8-15, GPT2-12-12, GPT2-16-10, GPT2-20-9, with similar parameter counts; size-2 models are GPT2-4-30, GPT2-8-21, GPT2-12-17, GPT2-16-15, GPT2-20-13, approximately twice the size of size-1 models. From Figure 9 , we observe that a 4-layer transformer, even with 1920 hidden dimensions, underperforms on our math datasets. Conversely, deeper but smaller models, such as a 20-layer 576-dim, perform very well. Comparing accuracies vertically reveals a clear correlation between model depth and performance. Thus, we infer that depth is likely essential for reasoning tasks, such as solving grade-school math problems.
Next, we try to reveal “why” this happens. We delved into how depth influences math problem-solving skills through the nece probing task, focusing on necessary parameters at distance t t from the query parameter, for t ∈ { 1 , 2 , … , 8 } t\in\{1,2,\dots,8\} . These parameters all have nece ( A ) = true \texttt{nece}(A)=\mathrm{true} , but we can probe the model to see how correct they are at predicting nece ( A ) \texttt{nece}(A) at different hidden layers.
Figure 10 shows our result. It reveals a correlation between the model’s layer hierarchy, reasoning accuracy, and mental reasoning depth . Shallower layers excel at predicting nece ( A ) \texttt{nece}(A) for parameters A A closer to the query, whereas deeper layers are more accurate and can predict nece ( A ) \texttt{nece}(A) for parameters further from the query. This suggests that the model employs layer-by-layer reasoning during the planning phase to recursively identify all parameters the query depends on, and:
Result 8 ( Figure 10 + 14 ) . The depth of a language model is crucial, likely due to the complexity of its hidden (mental) reasoning processes. A t t -step mental reasoning, such as mentally computing nece ( A ) \texttt{nece}(A) for parameters A A that are a distance t t from the query, may require deeper models for larger t t , assuming all other hyperparameters remain constant.
We make two disclaimers here. First, if the “backward thinking process” is added as CoT to the data (see the end of Section 4.2 ), then deep mental thinking is no longer required, reducing the language model’s depth requirement. However, in practice, many such “thinking processes” may not be included in standard math solutions or languages in general.
Second, the above claim does not imply that “a t t -step mental thinking requires a depth- t t transformer”. It is plausible for a single transformer layer (containing many sub-layers) to implement t > 1 t>1 mental thinking steps, though possibly with reduced accuracy as t t increases. We refrain from providing an exact correlation in this paper, as it heavily depends on the data distribution.
## 7 Conclusion
We use a synthetic setting to demonstrate that language models can learn to solve grade-school math problems through true generalization, rather than relying on data contamination or template memorization. We develop probing techniques to examine the models’ hidden reasoning processes. Our findings reveal that these models can learn math skills aligned with human cognitive processes, as well as “new thinking processes” not present in the training data. Additionally, we propose a method to predict a model’s errors before it begins to solve a problem and to explain why models make mistakes when they occur. Based on this discovery, we write a separate paper to improve language models’ math reasoning accuracy [ 21 ] . We also provide a principled approach to connect the model’s depth to its capable reasoning length. We believe this research opens doors to study the mathematical reasoning skills of language models from a different angle compared to pushing math benchmarks.
One may argue that iGSM may be very different from the pretrain data that modern LLMs use. While this may be true, we are looking into the future. Recall, even GPT-4/4o of today cannot few-shot learn to solve iGSM-med op = 11 \textsf{iGSM-med}^{\textsf{op}=11} (see Figure 2 ). From this perspective, it is reasonable to believe that future versions of LLMs will rely on synthetic math pretrain data to improve their reasoning skills. While one may not directly use iGSM , it is tempting to use existing LLMs (such as Llama-3) to turn iGSM into more natural formats while keeping the logical chains. On the other hand, we have discovered that models trained purely on the iGSM data make similar mistakes compared to GPT-4/4o (see Section 5 and Appendix G ). This further confirms that our findings do connect to practice, regarding the model’s hidden reasoning process.
Finally, Part 2 of this work series focuses on how language models solve grade-school math problems (including Part 2.2 [ 21 ] ). We also cover how language models learn language structures in Part 1 [ 1 ] (in particular, how they mentally perform dynamical programming), and learn world knowledge in Part 3 [ 3 , 2 , 4 ] .
Appendix
## Appendix A Result 1 — An Example in iGSM-hard with op = 21 \textsf{op}=21
## Appendix B Results 4-5 — Details on V-probing
Recall that we wish to conduct probing at the end of the problem description for the nece and dep tasks (before the solution for nece ; before the solution or even the question for dep ). For other tasks, we probe at the end of every solution sentence (including the start of the first solution sentence). The goal is to freeze a pretrained language model, then introduce a very small number of additional trainable parameters on top of it, and finetune them for each probing task.
Specifically, we take a pretrained language model, e.g., pretrained from the iGSM-hard training data. We freeze its parameters completely except for adding a trainable rank- r r update on the embedding layer to account for the task change (from next-token prediction to probing). Throughout this paper we use a small value r = 8 r=8 . We feed this network with training data that are the same as iGSM-hard , but truncated at exactly the position we wish to probe. Importantly, we append such inputs with a special starting token [START] along with a parameter name (or two names, if it is the dep ( A , B ) \texttt{dep}(A,B) task). We then extract the hidden states of the last token position at the last transformer layer, and add a trainable linear layer (a.k.a. linear head) to perform classification for one of the six probing tasks.
This probing method is illustrated in Figure 13 . We call it V(ariable)-Probing, because it can take an arbitrary number of variables (i.e., parameters in this paper) to allow us to perform functional probing inside the transformer.