Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure — part 4/4

Cleaned-text code points [46617, 49193). Bibliography omitted here; source.md includes it.

Errors of LLMs. We examine error cases of LLMs to understand their failure modes. For o3-Mini-High (which does not return the thinking tokens), the model summarizes the thinking processing at a high level with statements such as “Every acceptable solution of the many equations forces…” and “One may check by solving the huge simultaneous network of sum‐equations that…” , and hence it is difficult to pinpoint the specific errors. For Gemini-2.5-Pro, we find that the model never makes calculation errors, and among 10 10 random error examples, 7 7 stem from making (wrong) guesses without thorough consistency checks, 2 2 are caused by hallucinating unprovided facts, and 1 1 from a copy error. Overall, LLMs seem to struggle with forming integrated/compressed representations of information provided in context, and have to rely on extensive explicit search to recognize the connections between different pieces of information. With well-designed parametric memory, on the other hand, the facts could be more tightly connected and integrated, which enables models to solve the challenge with better performance and milder performance drop as problem scales increase.

## Appendix D Use of Large Language Models

Large Language Models (LLMs) were used solely as a general-purpose writing aid in the preparation of this paper. Specifically, they were used to help polish grammar and improve the clarity of certain sentences. No LLMs were used for research ideation, experimental design, data analysis, or drawing conclusions. All substantive contributions to the research and writing were made by the authors.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .