Less is More: Recursive Reasoning with Tiny Networks — part 1/3

Cleaned-text code points [0, 15551). Bibliography omitted here; source.md includes it.

# Less is More: Recursive Reasoning with Tiny Networks

###### Abstract

Hierarchical Reasoning Model (HRM) is a novel approach using two small neural networks recursing at different frequencies. This biologically inspired method beats Large Language models (LLMs) on hard puzzle tasks such as Sudoku, Maze, and ARC-AGI while trained with small models (27M parameters) on small data ( ∼ \sim 1000 examples). HRM holds great promise for solving hard problems with small networks, but it is not yet well understood and may be suboptimal. We propose Tiny Recursive Model (TRM), a much simpler recursive reasoning approach that achieves significantly higher generalization than HRM, while using a single tiny network with only 2 layers. With only 7M parameters, TRM obtains 45% test-accuracy on ARC-AGI-1 and 8% on ARC-AGI-2, higher than most LLMs (e.g., Deepseek R1, o3-mini, Gemini 2.5 Pro) with less than 0.01% of the parameters.

###### Keywords:

Alexia Jolicoeur-Martineau Samsung SAIL Montréal alexia.j@samsung.com

## 1 Introduction

While powerful, Large Language models (LLMs) can struggle on hard question-answer problems. Given that they generate their answer auto-regressively, there is a high risk of error since a single incorrect token can render an answer invalid. To improve their reliability, LLMs rely on Chain-of-thoughts (CoT) ( Wei et al., 2022 ) and Test-Time Compute (TTC) ( Snell et al., 2024 ) . CoTs seek to emulate human reasoning by having the LLM to sample step-by-step reasoning traces prior to giving their answer. Doing so can improve accuracy, but CoT is expensive, requires high-quality reasoning data (which may not be available), and can be brittle since the generated reasoning may be wrong. To further improve reliability, test-time compute can be used by reporting the most common answer out of K K or the highest-reward answer ( Snell et al., 2024 ) .

However, this may not be enough. LLMs with CoT and TTC are not enough to beat every problem. While LLMs have made significant progress on ARC-AGI ( Chollet, 2019 ) since 2019, human-level accuracy still has not been reached (6 years later, as of writing of this paper). Furthermore, LLMs struggle on the newer ARC-AGI-2 (e.g., Gemini 2.5 Pro only obtains 4.9% test accuracy with a high amount of TTC) ( Chollet et al., 2025 ; ARC Prize Foundation, 2025b ) .

An alternative direction has recently been proposed by Wang et al. (2025) . They propose a new way forward through their novel Hierarchical Reasoning Model (HRM), which obtains high accuracy on puzzle tasks where LLMs struggle to make a dent (e.g., Sudoku solving, Maze pathfinding, and ARC-AGI). HRM is a supervised learning model with two main novelties: 1) recursive hierarchical reasoning , and 2) deep supervision .

Recursive hierarchical reasoning consists of recursing multiple times through two small networks ( f L f_{L} at high frequency and f H f_{H} at low frequency) to predict the answer. Each network generates a different latent feature: f L f_{L} outputs z H z_{H} and f H f_{H} outputs z L z_{L} . Both features ( z L , z H z_{L},z_{H} ) are used as input to the two networks. The authors provide some biological arguments in favor of recursing at different hierarchies based on the different temporal frequencies at which the brains operate and hierarchical processing of sensory inputs.

Deep supervision consists of improving the answer through multiple supervision steps while carrying the two latent features as initialization for the improvement steps (after detaching them from the computational graph so that their gradients do not propagate). This provide residual connections, which emulates very deep neural networks that are too memory expensive to apply in one forward pass.

An independent analysis on the ARC-AGI benchmark showed that deep supervision seems to be the primary driver of the performance gains ( ARC Prize Foundation, 2025a ) . Using deep supervision doubled accuracy over single-step supervision (going from 19 % 19\% to 39 % 39\% accuracy), while recursive hierarchical reasoning only slightly improved accuracy over a regular model with a single forward pass (going from 35.7 % 35.7\% to 39.0 % 39.0\% accuracy). This suggests that reasoning across different supervision steps is worth it, but the recursion done in each supervision step is not particularly important.

In this work, we show that the benefit from recursive reasoning can be massively improved, making it much more than incremental. We propose Tiny Recursive Model (TRM), an improved and simplified approach using a much smaller tiny network with only 2 layers that achieves significantly higher generalization than HRM on a variety of problems. In doing so, we improve the state-of-the-art test accuracy on Sudoku-Extreme from 55% to 87%, Maze-Hard from 75% to 85%, ARC-AGI-1 from 40% to 45%, and ARC-AGI-2 from 5% to 8%.

## 2 Background

HRM is described in Algorithm 2 . We discuss the details of the algorithm further below.

### 2.1 Structure and goal

The focus of HRM is supervised learning. Given an input, produce an output. Both input and output are assumed to have shape [ B , L ] [B,L] (when the shape differs, padding tokens can be added), where B B is the batch-size and L L is the context-length.

HRM contains four learnable components: the input embedding f I ​ ( ⋅ , θ I ) f_{I}(\cdot;\theta_{I}) , low-level recurrent network f L ​ ( ⋅ , θ L ) f_{L}(\cdot;\theta_{L}) , high-level recurrent network f H ​ ( ⋅ , θ H ) f_{H}(\cdot;\theta_{H}) , and the output head f O ​ ( ⋅ , θ O ) f_{O}(\cdot;\theta_{O}) . Once the input is embedded, the shape becomes [ B , L , D ] [B,L,D] where D D is the embedding size. Each network is a 4-layer Transformers architecture ( Vaswani et al., 2017 ) , with RMSNorm ( Zhang & Sennrich, 2019 ) , no bias ( Chowdhery et al., 2023 ) , rotary embeddings ( Su et al., 2024 ) , and SwiGLU activation function ( Hendrycks & Gimpel, 2016 ; Shazeer, 2020 ) .

### 2.2 Recursion at two different frequencies

Given the hyperparameters used by Wang et al. (2025) ( n = 2 n=2 f L f_{L} steps, 1 f H f_{H} steps; done T = 2 T=2 times), a forward pass of HRM is done as follows: x \displaystyle x ← f I ​ ( x ~ ) \displaystyle\leftarrow f_{I}(\tilde{x}) z L \displaystyle z_{L} ← f L ​ ( z L + z H + x ) ​ # without gradients \displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)\hskip 6.0pt\text{\# without gradients} z L \displaystyle z_{L} ← f L ​ ( z L + z H + x ) ​ # without gradients \displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)\hskip 6.0pt\text{\# without gradients} z H \displaystyle z_{H} ← f H ( z L + z H ) # without gradients \displaystyle\leftarrow f_{H}\left(z_{L}+z_{H}\right)\hskip 22.0pt\text{\# without gradients} z L \displaystyle z_{L} ← f L ​ ( z L + z H + x ) ​ # without gradients \displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)\hskip 6.0pt\text{\# without gradients} z L \displaystyle z_{L} ← z L . d ​ e ​ t ​ a ​ c ​ h ​ ( ) \displaystyle\leftarrow z_{L}.detach() z H \displaystyle z_{H} ← z H . d ​ e ​ t ​ a ​ c ​ h ​ ( ) \displaystyle\leftarrow z_{H}.detach() z L \displaystyle z_{L} ← f L ​ ( z L + z H + x ) ​ # with gradients \displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right)\hskip 6.0pt\text{\# with gradients} z H \displaystyle z_{H} ← f H ( z L + z H ) # with gradients \displaystyle\leftarrow f_{H}\left(z_{L}+z_{H}\right)\hskip 22.0pt\text{\# with gradients} y ^ \displaystyle\hat{y} ← argmax ​ ( f O ​ ( z H ) ) \displaystyle\leftarrow\text{argmax}(f_{O}\left(z_{H}\right))

where y ^ \hat{y} is the predicted output answer, z L z_{L} and z H z_{H} are either initialized embeddings or the embeddings of the previous deep supervision step (after detaching them from the computational graph). As can be seen, a forward pass of HRM consists of applying 6 function evaluations, where the first 4 function evaluations are detached from the computational graph and are not back-propagated through. The authors uses n = 2 n=2 with T = 2 T=2 in all experiments, but HRM can be generalized by allowing for an arbitrary number of L steps ( n n ) and recursions ( T T ) as shown in Algorithm 2 .

### 2.3 Fixed-point recursion with 1-step gradient approximation

Assuming that ( z L z_{L} , z H z_{H} ) reaches a fixed-point ( z L ∗ z_{L}^{*} , z H ∗ z_{H}^{*} ) through recursing from both f L f_{L} and f H f_{H} , z L ∗ \displaystyle z_{L}^{*} ≈ f L ​ ( z L ∗ + z H + x ) \displaystyle\approx f_{L}\left(z_{L}^{*}+z_{H}+x\right) z H ∗ \displaystyle z_{H}^{*} ≈ f H ​ ( z L + z H ∗ ) , \displaystyle\approx f_{H}\left(z_{L}+z_{H}^{*}\right), the Implicit Function Theorem ( Krantz & Parks, 2002 ) with the 1-step gradient approximation ( Bai et al., 2019 ) is used to approximate the gradient by back-propagating only the last f L f_{L} and f H f_{H} steps. This theorem is used to justify only tracking the gradients of the last two steps (out of 6), which greatly reduces memory demands.

### 2.4 Deep supervision

To improve effective depth, deep supervision is used. This consists of reusing the previous latent features ( z H z_{H} and z L z_{L} ) as initialization for the next forward pass. This allows the model to reason over many iterations and improve its latent features ( z L z_{L} and z H z_{H} ) until it (hopefully) converges to the correct solution. At most N s ​ u ​ p = 16 N_{sup}=16 supervision steps are used.

### 2.5 Adaptive computational time (ACT)

With deep supervision, each mini-batch of data samples must be used for N s ​ u ​ p = 16 N_{sup}=16 supervision steps before moving to the next mini-batch. This is expensive, and there is a balance to be reached between optimizing a few data examples for many supervision steps versus optimizing many data examples with less supervision steps. To reach a better balance, a halting mechanism is incorporated to determine whether the model should terminate early. It is learned through a Q-learning objective that requires passing the z H z_{H} through an additional head and running an additional forward pass (to determine if halting now rather than later would have been preferable). They call this method Adaptive computational time (ACT). It is only used during training, while the full N s ​ u ​ p = 16 N_{sup}=16 supervision steps are done at test time to maximize downstream performance. ACT greatly diminishes the time spent per example (on average spending less than 2 steps on the Sudoku-Extreme dataset rather than the full N s ​ u ​ p = 16 N_{sup}=16 steps), allowing more coverage of the dataset given a fixed number of training iterations.

### 2.6 Deep supervision and 1-step gradient approximations replaces BPTT

Deep supervision and the 1-step gradient approximation provide a more biologically plausible and less computationally-expansive alternative to Backpropagation Through Time (BPTT) ( Werbos, 1974 ; Rumelhart et al., 1985 ; LeCun, 1985 ) for solving the temporal credit assignment (TCA) ( Rumelhart et al., 1985 ; Werbos, 1988 ; Elman, 1990 ) problem ( Lillicrap & Santoro, 2019 ) . The implication is that HRM can learn what would normally require an extremely large network without having to back-propagate through its entire depth. Given the hyperparameters used by Jang et al. (2023) in all their experiments, HRM effectively reasons over n l ​ a ​ y ​ e ​ r ​ s ​ ( n + 1 ) ​ T ​ N s ​ u ​ p = 4 ∗ ( 2 + 1 ) ∗ 2 ∗ 16 = 384 n_{layers}(n+1)TN_{sup}=4*(2+1)*2*16=384 layers of effective depth.

### 2.7 Summary of HRM

HRM leverages recursion from two networks at different frequencies (high frequency versus low frequency) and deep supervision to learn to improve its answer over multiple supervision steps (with ACT to reduce time spent per data example). This enables the model to imitate extremely large depth without requiring backpropagation through all layers. This approach obtains significantly higher performance on hard question-answer tasks that regular supervised models struggle with. However, this method is quite complicated, relying a bit too heavily on uncertain biological arguments and fixed-point theorems that are not guaranteed to be applicable. In the next section, we discuss those issues and potential targets for improvements in HRM.

## 3 Target for improvements in Hierarchical Reasoning Models

In this section, we identify key targets for improvements in HRM, which will be addressed by our proposed method, Tiny Recursion Models (TRMs).

### 3.1 Implicit Function Theorem (IFT) with 1-step gradient approximation

HRM only back-propagates through the last 2 of the 6 recursions. The authors justify this by leveraging the Implicit Function Theorem (IFT) and one-step approximation, which states that when a recurrent function converges to a fixed point, backpropagation can be applied in a single step at that equilibrium point.

There are concerns about applying this theorem to HRM. Most importantly, there is no guarantee that a fixed-point is reached. Deep equilibrium models normally do fixed-point iteration to solve for the fixed point z ∗ = f ⁡ ( z ∗ ) z^{*}=f(z^{*}) ( Bai et al., 2019 ) . However, in the case of HRM, it is not iterating to the fixed-point but simply doing forward passes of f L f_{L} and f H f_{H} . To make matters worse, HRM is only doing 4 recursions before stopping to apply the one-step approximation. After its first loop of two f L f_{L} and 1 f H f_{H} evaluations, it only apply a single f L f_{L} evaluation before assuming that a fixed-point is reached for both z L z_{L} and z H z_{H} ( z L ∗ = f L ​ ( z L ∗ + z H + x ) z_{L}^{*}=f_{L}(z_{L}^{*}+z_{H}+x) and z H ∗ = f H ​ ( z L ∗ + z H ∗ ) z_{H}^{*}=f_{H}(z_{L}^{*}+z_{H}^{*}) ). Then, the one-step gradient approximation is applied to both latent variables in succession.

The authors justify that a fixed-point is reached by depicting an example with n = 7 n=7 and T = 7 T=7 where the forward residuals is reduced over time (Figure 3 in Wang et al. (2025) ). Even in this setting, which is different from the much smaller n = 2 n=2 and T = 2 T=2 used in every experiment of their paper, we observe the following: 1. the residual for z H z_{H} is clearly well above 0 at every step

2. the residual for z L z_{L} only becomes closer to 0 after many cycles, but it remains significantly above 0

3. z L z_{L} is very far from converged after one f L f_{L} evaluation at T T cycles, which is when the fixed-point is assumed to be reached and the 1-step gradient approximation is used

Thus, while the application of the IFT theorem and 1-step gradient approximation to HRM has some basis since the residuals do generally reduce over time, a fixed point is unlikely to be reached when the theorem is actually applied.

In the next section, we show that we can bypass the need for the IFT theorem and 1-step gradient approximation, thus bypassing the issue entirely.

### 3.2 Twice the forward passes with Adaptive computational time (ACT)

HRM uses Adaptive computational time (ACT) during training to optimize the time spent of each data sample. Without ACT, N s ​ u ​ p = 16 N_{sup}=16 supervision steps would be spent on the same data sample, which is highly inefficient. They implement ACT through an additional Q-learning objective, which decides when to halt and move to a new data sample rather than keep iterating on the same data. This allows much more efficient use of time especially since the average number of supervision steps during training is quite low with ACT (less than 2 steps on the Sudoku-Extreme dataset as per their reported numbers).