Less is More: Recursive Reasoning with Tiny Networks — part 2/3

Cleaned-text code points [15551, 31419). Bibliography omitted here; source.md includes it.

However, ACT comes at a cost. This cost is not directly shown in the HRM’s paper, but it is shown in their official code. The Q-learning objective relies on a halting loss and a continue loss. The continue loss requires an extra forward pass through HRM (with all 6 function evaluations). This means that while ACT optimizes time more efficiently per sample, it requires 2 forward passes per optimization step. The exact formulation is shown in Algorithm 2 .

In the next section, we show that we can bypass the need for two forward passes in ACT.

### 3.3 Hierarchical interpretation based on complex biological arguments

The HRM’s authors justify the two latent variables and two networks operating at different hierarchies based on biological arguments, which are very far from artificial neural networks. They even try to match HRM to actual brain experiments on mice. While interesting, this sort of explanation makes it incredibly hard to parse out why HRM is designed the way it is. Given the lack of ablation table in their paper, the over-reliance on biological arguments and fixed-point theorems (that are not perfectly applicable), it is hard to determine what parts of HRM is helping what and why. Furthermore, it is not clear why they use two latent features rather than other combinations of features.

In the next section, we show that the recursive process can be greatly simplified and understood in a much simpler manner that does not require any biological argument, fixed-point theorem, hierarchical interpretation, nor using two networks. It also explains why 2 is the optimal number of features ( z L z_{L} and z H z_{H} ).

## 4 Tiny Recursion Models

In this section, we present Tiny Recursion Models (TRMs). Contrary to HRM, TRM requires no complex mathematical theorem, hierarchy, nor biological arguments. It generalizes better while requiring only a single tiny network (instead of two medium-size networks) and a single forward pass for the ACT (instead of 2 passes). Our approach is described in Algorithm 3 and illustrated in Figure 1 . We also provide an ablation in Table 1 on the Sudoku-Extreme dataset (a dataset of difficult Sudokus with only 1K training examples, but 423K test examples). Below, we explain the key components of TRMs.

### 4.1 No fixed-point theorem required

HRM assumes that the recursions converge to a fixed-point for both z L z_{L} and z H z_{H} in order to leverage the 1-step gradient approximation ( Bai et al., 2019 ) . This allows the authors to justify only back-propagating through the last two function evaluations (1 f L f_{L} and 1 f H f_{H} ). To bypass this theoretical requirement, we define a full recursion process as containing n n evaluations of f L f_{L} and 1 evaluation of f H f_{H} : z L \displaystyle z_{L} ← f L ​ ( z L + z H + x ) \displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right) \displaystyle... z L \displaystyle z_{L} ← f L ​ ( z L + z H + x ) \displaystyle\leftarrow f_{L}\left(z_{L}+z_{H}+x\right) z H \displaystyle z_{H} ← f H ​ ( z L + z H ) . \displaystyle\leftarrow f_{H}\left(z_{L}+z_{H}\right). Then, we simply back-propagate through the full recursion process.

Through deep supervision, the models learns to take any ( z L , z H ) (z_{L},z_{H}) and improve it through a full recursion process, hopefully making z H z_{H} closer to the solution. This means that by the design of the deep supervision goal, running a few full recursion processes (even without gradients) is expected to bring us closer to the solution. We propose to run T − 1 T-1 recursion processes without gradient to improve ( z L , z H ) (z_{L},z_{H}) before running one recursion process with backpropagation.

Thus, instead of using the 1-step gradient approximation, we apply a full recursion process containing n n evaluations of f L f_{L} and 1 evaluation of f H f_{H} . This removes entirely the need to assume that a fixed-point is reached and the use of the IFT theorem with 1-step gradient approximation. Yet, we can still leverage multiple backpropagation-free recursion processes to improve ( z L , z H ) (z_{L},z_{H}) . With this approach, we obtain a massive boost in generalization on Sudoku-Extreme (improving TRM from 56.5% to 87.4%; see Table 1).

### 4.2 Simpler reinterpretation of z H z_{H} and z L z_{L}

HRM is interpreted as doing hierarchical reasoning over two latent features of different hierarchies due to arguments from biology. However, one might wonder why use two latent features instead of 1, 3, or more? And do we really need to justify these so-called ”hierarchical” features based on biology to make sense of them? We propose a simple non-biological explanation, which is more natural, and directly answers the question of why there are 2 features.

The fact of the matter is: z H z_{H} is simply the current (embedded) solution. The embedding is reversed by applying the output head and rounding to the nearest token using the argmax operation. On the other hand, z L z_{L} is a latent feature that does not directly correspond to a solution, but it can be transformed into a solution by applying z H ← f H ​ ( x , z L , z H ) z_{H}\leftarrow f_{H}(x,z_{L},z_{H}) . We show an example on Sudoku-Extreme in Figure 6 to highlight the fact that z H z_{H} does correspond to the solution, but z L z_{L} does not.

Once this is understood, hierarchy is not needed; there is simply an input x x , a proposed solution y y (previously called z H z_{H} ), and a latent reasoning feature z z (previously called z L z_{L} ). Given the input question x x , current solution y y , and current latent reasoning z z , the model recursively improves its latent z z . Then, given the current latent z z and the previous solution y y , the model proposes a new solution y y (or stay at the current solution if its already good).

Although this has no direct influence on the algorithm, this re-interpretation is much simpler and natural. It answers the question about why two features: remembering in context the question x x , previous reasoning z z , and previous answer y y helps the model iterate on the next reasoning z z and then the next answer y y . If we were not passing the previous reasoning z z , the model would forget how it got to the previous solution y y (since z z acts similarly as a chain-of-thought). If we were not passing the previous solution y y , then the model would forget what solution it had and would be forced to store the solution y y within z z instead of using it for latent reasoning. Thus, we need both y y and z z separately, and there is no apparent reason why one would need to split z z into multiple features.

While this is intuitive, we wanted to verify whether using more or less features could be helpful. Results are shown in Table 2 .

More features ( > 2 >2 ): We tested splitting z z into different features by treating each of the n n recursions as producing a different z i z_{i} for i = 1 , … , n i=1,...,n . Then, each z i z_{i} is carried across supervision steps. The approach is described in Algorithm 5 . In doing so, we found performance to drop. This is expected because, as discussed, there is no apparent need for splitting z z into multiple parts. It does not have to be hierarchical.

Single feature : Similarly, we tested the idea of taking a single feature by only carrying z H z_{H} across supervision steps. The approach is described in Algorithm 4 . In doing so, we found performance to drop. This is expected because, as discussed, it forces the model to store the solution y y within z z .

Thus, we explored using more or less latent variables on Sudoku-Extreme, but found that having only y y and z z lead to better test accuracy in addition to being the simplest more natural approach.

### 4.3 Single network

HRM uses two networks, one applied frequently as a low-level module f H f_{H} and one applied rarely as an high-level module ( f H f_{H} ). This requires twice the number of parameters compared to regular supervised learning with a single network.

As mentioned previously, while f L f_{L} iterates on the latent reasoning feature z z ( z L z_{L} in HRM), the goal of f H f_{H} is to update the solution y y ( z H z_{H} in HRM) given the latent reasoning and current solution. Importantly, since z ← f L ​ ( x + y + z ) z\leftarrow f_{L}(x+y+z) contains x x but y ← f H ​ ( y + z ) y\leftarrow f_{H}(y+z) does not contains x x , the task to achieve (iterating on z z versus using z z to update y y ) is directly specified by the inclusion or lack of x x in the inputs. Thus, we considered the possibility that both networks could be replaced by a single network doing both tasks. In doing so, we obtain better generalization on Sudoku-Extreme (improving TRM from 82.4% to 87.4%; see Table 1) while reducing the number of parameters by half. It turns out that a single network is enough.

### 4.4 Less is more

We attempted to increase capacity by increasing the number of layers in order to scale the model. Surprisingly, we found that adding layers decreased generalization due to overfitting. In doing the opposite, decreasing the number of layers while scaling the number of recursions ( n n ) proportionally (to keep the amount of compute and emulated depth approximately the same), we found that using 2 layers (instead of 4 layers) maximized generalization. In doing so, we obtain better generalization on Sudoku-Extreme (improving TRM from 79.5% to 87.4%; see Table 1) while reducing the number of parameters by half (again).

It is quite surprising that smaller networks are better, but 2 layers seems to be the optimal choice. Bai & Melas-Kyriazi (2024) also observed optimal performance for 2-layers in the context of deep equilibrium diffusion models; however, they had similar performance to the bigger networks, while we instead observe better performance with 2 layers. This may appear unusual, as with modern neural networks, generalization tends to directly correlate with model sizes. However, when data is too scarce and model size is large, there can be an overfitting penalty ( Kaplan et al., 2020 ) . This is likely an indication that there is too little data. Thus, using tiny networks with deep recursion and deep supervision appears to allow us to bypass a lot of the overfitting.

### 4.5 attention-free architecture for tasks with small fixed context length

Self-attention is particularly good for long-context lengths when L ≫ D L\gg D since it only requires a matrix of [ D , 3 ​ D ] [D,3D] parameters, even though it can account for the whole sequence. However, when focusing on tasks where L ≤ D L\leq D , a linear layer is cheap, requiring only a matrix of [ L , L ] [L,L] parameters. Taking inspiration from the MLP-Mixer ( Tolstikhin et al., 2021 ) , we can replace the self-attention layer with a multilayer perceptron (MLP) applied on the sequence length. Using an MLP instead of self-attention, we obtain better generalization on Sudoku-Extreme (improving from 74.7% to 87.4%; see Table 1 ). This worked well on Sudoku 9x9 grids, given the small and fixed context length; however, we found this architecture to be suboptimal for tasks with large context length, such as Maze-Hard and ARC-AGI (both using 30x30 grids). We show results with and without self-attention for all experiments.

### 4.6 No additional forward pass needed with ACT

As previously mentioned, the implementation of ACT in HRM through Q-learning requires two forward passes, which slows down training. We propose a simple solution, which is to get rid of the continue loss (from the Q-learning) and only learn a halting probability through a Binary-Cross-Entropy loss of having reached the correct solution. By removing the continue loss, we remove the need for the expensive second forward pass, while still being able to determine when to halt with relatively good accuracy. We found no significant difference in generalization from this change (going from 86.1% to 87.4%; see Table 1 ).

### 4.7 Exponential Moving Average (EMA)

On small data (such as Sudoku-Extreme and Maze-Hard), HRM tends to overfit quickly and then diverge. To reduce this problem and improves stability, we integrate Exponential Moving Average (EMA) of the weights, a common technique in GANs and diffusion models to improve stability ( Brock et al., 2018 ; Song & Ermon, 2020 ) . We find that it prevents sharp collapse and leads to higher generalization (going from 79.9% to 87.4%; see Table 1 ).

### 4.8 Optimal the number of recursions

We experimented with different number of recursions by varying T T and n n and found that T = 3 , n = 3 T=3,n=3 (equivalent to 48 recursions) in HRM and T = 3 , n = 6 T=3,n=6 in TRM (equivalent to 42 recursions) to lead to optimal generalization on Sudoku-Extreme. More recursions could be helpful for harder problems (we have not tested it, given our limited resources); however, increasing either T T or n n incurs massive slowdowns. We show results at different n n and T T for HRM and TRM in Table 3 . Note that TRM requires backpropagation through a full recursion process, thus increasing n n too much leads to Out Of Memory (OOM) errors. However, this memory cost is well worth its price in gold.

In the following section, we show our main results on multiple datasets comparing HRM, TRM, and LLMs.

## 5 Results

Following Wang et al. (2025) , we test our approach on the following datasets: Sudoku-Extreme ( Wang et al., 2025 ) , Maze-Hard ( Wang et al., 2025 ) , ARC-AGI-1 ( Chollet, 2019 ) and, ARC-AGI-2 ( Chollet et al., 2025 ) . Results are presented in Tables 4 and 5 . Hyperparameters are detailed in Section Hyper-parameters and setup . Datasets are discussed below.

Sudoku-Extreme consists of extremely difficult Sudoku puzzles ( Dillion, 2025 ; Palm et al., 2018 ; Park, 2018 ) (9x9 grid), for which only 1K training samples are used to test small-sample learning. Testing is done on 423K samples. Maze-Hard consists of 30x30 mazes generated by the procedure by Lehnert et al. (2024) whose shortest path is of length above 110; both the training set and test set include 1000 mazes.

ARC-AGI-1 and ARC-AGI-2 are geometric puzzles involving monetary prizes. Each puzzle is designed to be easy for a human, yet hard for current AI models. Each puzzle task consists of 2-3 input–output demonstration pairs and 1-2 test inputs to be solved. The final score is computed as the accuracy over all test inputs from two attempts to produce the correct output grid. The maximum grid size is 30x30. ARC-AGI-1 contains 800 tasks, while ARC-AGI-2 contains 1120 tasks. We also augment our data with the 160 tasks from the closely related ConceptARC dataset ( Moskvichev et al., 2023 ) . We provide results on the public evaluation set for both ARC-AGI-1 and ARC-AGI-2.

While these datasets are small, heavy data-augmentation is used in order to improve generalization. Sudoku-Extreme uses 1000 shuffling (done without breaking the Sudoku rules) augmentations per data example. Maze-Hard uses 8 dihedral transformations per data example. ARC-AGI uses 1000 data augmentations (color permutation, dihedral-group, and translations transformations) per data example. The dihedral-group transformations consist of random 90-degree rotations, horizontal/vertical flips, and reflections.

From the results, we see that TRM without self-attention obtains the best generalization on Sudoku-Extreme (87.4% test accuracy). Meanwhile, TRM with self-attention generalizes better on the other tasks (probably due to inductive biases and the overcapacity of the MLP on large 30x30 grids). TRM with self-attention obtains 85.3% accuracy on Maze-Hard, 44.6% accuracy on ARC-AGI-1, and 7.8% accuracy on ARC-AGI-2 with 7M parameters. This is significantly higher than the 74.5%, 40.3%, and 5.0% obtained by HRM using 4 times the number of parameters (27M).

## 6 Conclusion