Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — part 4/4

Cleaned-text code points [46688, 50136). Bibliography omitted here; source.md includes it.

We share the parameters of the first 4 4 layers and the last 4 4 layers, similar as in Universal Transformer [ 7 ] . This would allow the model to share the knowledge in the upper and lower layers. The results on the setting with | ℰ | = 2000 |\mathcal{E}|=2000 and ϕ = 12.6 \phi=12.6 are shown in Figure 14 . It could be seen that the parameter-sharing scheme can unlock OOD generalization, even though it is gained much more slowly than ID generalization during grokking.

### E.3 Comparison task across different inferred/atomic ratio

Figure 15 includes the result for ϕ ∈ { 3.6 , 7.2 , 9.0 , 12.6 } \phi\in\{3.6,7.2,9.0,12.6\} for the comparison task. It can be seen that a higher ratio ϕ \phi would give a higher generalization speed, consistent with the results in the composition task.

## Appendix F Complex Reasoning Task

For the complex reasoning task, for each attribute, we include 3 3 % random facts from the comparisons between ID and OOD entities and the comparisons between ID and ID entities. In total, the training set contains 18 18 K ID atomic facts, 437 437 K (ID, ID) comparisons, and 108 108 K (ID, OOD) comparisons, altogether 563 563 K facts ( 28 28 K on average for each attribute). For translating the facts into natural language for testing LLMs with non-parametric memory, we always use the attribute age 13 13 13 Recall that we test each attribute separately by grouping the facts/queries. (we find that the choice of attribute does not exhibit notable impact) and the templates “The age of {entity} is {attribute value}.” and “{entity 1 1 } is {younger than/older than/in the same age as} {entity 2 2 }.’’ for atomic facts and comparisons. The entities are mapped to distinct random names generated by a random generator. 14 14 14 https://pypi.org/project/names/ We also try different mappings (e.g., unique IDs) and templates, and find the results to be consistent. All (retrieved) facts are randomly permuted and concatenated before being loaded into the LLM context.

The train/test accuracy, and also the accuracy of inferring the attribute values of the query entities (which we test using the same format as the atomic facts in training) are included in Figure 16 . It could be seen that, during grokking, the model gradually locates the ground truth attribute values of the query entities (note that the model is not explicitly encouraged or trained to do this), allowing the model to solve the problem efficiently with near-perfect accuracy.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .