Reasoning on the Simplex: Geometric Fixed-Point Models — part 1/4

Cleaned-text code points [0, 15871). Bibliography omitted here; source.md includes it.

# Reasoning on the Simplex: Geometric Fixed-Point Models

###### Abstract

Looped reasoners spend test-time compute by iterating a weight-tied map, but a small residual does not mean the state is a fixed point when that map lives in unconstrained latent space. We propose Geometric Fixed-Point Reasoning (GFPR), in which the iterated state is the prediction itself: a field of categorical beliefs on a product of simplices, whose arg ⁡ max \arg\max is the answer at every step. Because the state is a belief, task structure can be imposed through compact convex relaxations, either as structured readouts or directly in the recurrent state; in the latter case the update remains a continuous self-map, so a fixed point exists for any parameters. At about 7M parameters, GFPR reaches 95.1% exact match on Sudoku-Extreme, 92.0% on Maze-Hard, and 100% sequence accuracy on S 5 S_{5} length 128, above the published FPRM numbers at the same scale. The same update also trains a 201M language model on FineWeb-Edu in which each site is a distribution over the vocabulary; with 24 Picard steps it is above GPT-2 small on four zero-shot multiple-choice tasks and above GPT-2 medium on ARC-Easy.

## 1 Introduction

Reasoning benchmarks often mix easy and hard instances, and the hard ones need more internal computation to solve reliably. A popular response is to add test-time compute without training a larger network, so hard examples can receive more inference steps ( Snell et al., 2024 ) . In large language models, one common mechanism is to generate a longer chain of thought before the final answer ( Wei et al., 2022 ; Kojima et al., 2022 ) . That works well on many open-ended math and logic prompts, but chain-of-thought reasoning is not always the right interface: it ties computation to autoregressive text, grows latency with every token, and offers no grid-shaped state on which to enforce local constraints ( Wang et al., 2025 ) .

A different way to spend test-time compute is to apply the same neural block again and again: keep a latent state and update it with a weight-tied map, often for far more steps at inference than during training ( Jolicoeur-Martineau, 2025 ; Wang et al., 2025 ; Movahedi et al., 2026 ; Huang et al., 2026 ) . Read this way, looped reasoning looks like fixed-point iteration: repeat an update until successive states agree, then decode an answer from the settled state ( Bai et al., 2019 ; Movahedi et al., 2026 ; Huang et al., 2026 ) . FPRM ( Movahedi et al., 2026 ) makes this explicit and stops when the iterate residual is small, unlike TRM/HRM-style ACT halting ( Graves, 2016 ; Jolicoeur-Martineau, 2025 ; Wang et al., 2025 ) .

Halting on a small residual only certifies an answer if the looped map has a fixed point that the iteration actually reaches. For an arbitrary latent vector in ℝ d \mathbb{R}^{d} the standard guarantee is a contractive map. Movahedi et al. (2026) show that small residual scaling is sufficient for contraction, note that it is not guaranteed in practice, and add a decaying damping step to suppress the resulting oscillations. On their public Sudoku checkpoint, however, the state at which inference halts is not a fixed point, and even long after the halt the one-step map is typically not locally contractive (Section 2 ).

We propose Geometric Fixed-Point Reasoning (GFPR), which keeps damped fixed-point iteration but changes the space in which the iterate lives. The state is a field of categorical beliefs, one probability vector per grid cell or token, and every update returns to this product of simplices through a softmax readout. After damped Picard iteration the answer is the arg ⁡ max \arg\max of those beliefs, with known outputs pinned throughout, and the state can be read at any step (Section 3 ). The same beliefs also support task-specific convex structure, used either as a structured readout or directly as the recurrent state (Section 3.4 ). When the recurrent state itself is constrained to a compact convex set, the update remains a continuous self-map, so a fixed point exists; which one the iteration reaches is decided by training (Section 3.2 ). At ∼ 7 {\sim}7 M parameters, GFPR reaches 95.1% exact match on Sudoku-Extreme and 92.0% on Maze-Hard, compared with 94.2% and 87.0% for FPRM at the same scale (Section 4 ). The same update, trained as a 201M language model on FineWeb-Edu, is compared with GPT-2 on standard zero-shot multiple-choice tasks (section 4.4 ).

To sum up, our contributions are: • We introduce GFPR, a looped reasoner whose state is a field of categorical beliefs and whose answer is the arg ⁡ max \arg\max of that state (section 3 ).

• We show that this state lets us impose structural constraints for classical reasoning tasks such as Sudoku, mazes, and permutations (section 3.4 ).

• We evaluate GFPR on Sudoku-Extreme, Maze-Hard, and S 5 S_{5} state tracking, where it is above the published FPRM numbers at similar scale (section 4 ).

• We train a 201M simplex language model on FineWeb-Edu and compare it with GPT-2 small and medium (section 4.4 ).

• We study the iteration dynamics of GFPR and compare them with other looped reasoners, showing that GFPR settles to a fixed point while FPRM typically halts at a state that is not one (section 2 ).

## 2 Background: Looped Reasoning as Fixed-Point Iteration

A looped reasoner applies one weight-tied map F θ ​ ( ⋅ , x ) F_{\theta}(\cdot\,;x) to a state u u . With damping β ∈ ( 0 , 1 ] \beta\in(0,1] , u k + 1 = T β ​ ( u k ) , T β ​ ( u ) = ( 1 − β ) ​ u + β ​ F θ ​ ( u , x ) , u_{k+1}=T_{\beta}(u_{k}),\qquad T_{\beta}(u)=(1-\beta)\,u+\beta\,F_{\theta}(u;x), (1) whose fixed points satisfy u ⋆ = F θ ​ ( u ⋆ , x ) u^{\star}=F_{\theta}(u^{\star};x) . FPRM ( Movahedi et al., 2026 ) halts on a small relative iterate residual with decaying step size; TRM and HRM use ACT or a step cap ( Graves, 2016 ; Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ) ; DEQs solve for u ⋆ u^{\star} implicitly ( Bai et al., 2019 ) .

#### Existence and reachability.

Existence of a fixed point and convergence of equation 1 are different questions. Contraction answers both: if F θ F_{\theta} is L L -Lipschitz with L < 1 L<1 , Banach’s theorem ( Banach, 1922 ; Ortega and Rheinboldt, 2000 ; Kelley, 1995 ) gives a unique fixed point and global convergence, and damping preserves 1 − β + β ​ L < 1 1-\beta+\beta L<1 . For a trained network one usually checks local stability at a putative fixed point u ⋆ u^{\star} through the Jacobian J F J_{F} : eigenvalues λ \lambda of J F J_{F} become 1 − β + β ​ λ 1-\beta+\beta\lambda under damping, and u ⋆ u^{\star} attracts nearby states when all lie in the unit disk. If ρ ⁡ ( J F ) < 1 \rho(J_{F})<1 this holds for every β \beta ; if ρ ⁡ ( J F ) > 1 \rho(J_{F})>1 , F θ F_{\theta} is not locally contractive in any norm ( Horn and Johnson, 2012 ) , yet damped iteration may still converge. Non-normal J F J_{F} can make the residual non-monotone, and a small halt residual need not place the state near u ⋆ u^{\star} when J F J_{F} has an eigenvalue close to one ( Trefethen and Embree, 2005 ; Kerg et al., 2019 ) .

Brouwer’s theorem ( Brouwer, 1911 ) guarantees a fixed point for every continuous self-map of a nonempty compact convex set, with no contraction and for every θ \theta , but not uniqueness or convergence. GFPR’s belief polytope is of this kind (section 3 ); training must make the correct equilibrium reachable from the uniform start.

#### Are FPRM’s halted states fixed points?

FPRM iterates a free latent y ∈ ℝ d y\in\mathbb{R}^{d} and uses pre-norm blocks, residual scaling, and decaying step size s s to encourage contraction ( Movahedi et al., 2026 ) . We run its public Sudoku checkpoint with the authors’ loop (relative residual threshold 0.1 0.1 , s ← 0.997 ​ s s\leftarrow 0.997s on stall; supplementary material).

On 1000 held-out Sudoku-Extreme puzzles, 89% are solved, yet no halted state has relative RMS residual below 10 − 3 10^{-3} (median 3.1 × 10 − 2 3.1\times 10^{-2} ). With halting disabled for 8000 steps, the same probe with K pi = 10 K_{\mathrm{pi}}{=}10 still reports ‖ J F ​ v ‖ > 1 \|J_{F}v\|>1 on 75% of 128 held-out puzzles (supplementary material); we have not cross-checked that fraction with Arnoldi, so it is a diagnostic of local expansion under the probe rather than a population claim that ρ ⁡ ( J F ) > 1 \rho(J_{F})>1 .

Figure 2 is a single-puzzle case study both models solve. GFPR’s residual reaches float64 precision (Arnoldi largest Ritz modulus ≈ 1.3 × 10 − 3 \approx 1.3\times 10^{-3} on this puzzle); FPRM’s stays near 10 − 2 10^{-2} through its halt at k = 36 k=36 . Near the halt the dominant Jacobian mode is the pair λ ≈ − 0.13 ± 1.03 ​ i \lambda\approx-0.13\pm 1.03i ( | λ | ≈ 1.04 |\lambda|\approx 1.04 ). Any step s < 0.966 s<0.966 would damp it, but the halt fires at s ≈ 0.99 s\approx 0.99 , where | 1 − s + s ​ λ | ≈ 1.03 |1-s+s\lambda|\approx 1.03 : the update FPRM applies at the state it returns still expands along this mode.

## 3 Method

### 3.1 Belief state and update

GFPR represents a problem with n n sites (grid cells or tokens) by a state p = ( p 1 , … , p n ) p=(p_{1},\dots,p_{n}) , where each p i p_{i} is a probability vector over K K output symbols and A A auxiliary coordinates. The output block is what the task decodes, for example digits in Sudoku; the auxiliary coordinates act as registers for intermediate computation and are never read out. Let 𝒦 \mathcal{K} be the set of sites whose output is given by the input (for example Sudoku clues), and let 𝒫 x \mathcal{P}_{x} be the product of simplices in which every site of 𝒦 \mathcal{K} is fixed to the one-hot vector of its given symbol. With damping β ∈ ( 0 , 1 ] \beta\in(0,1] , one step is F θ ​ ( p , x ) = Π x ​ ( softmax ⁡ ( f θ ​ ( W in ​ p + Enc ⁡ ( x ) ) ) ) , T β ​ ( p ) = ( 1 − β ) ​ p + β ​ F θ ​ ( p , x ) , F_{\theta}(p;\,x)=\Pi_{x}\Bigl(\mathrm{softmax}\bigl(f_{\theta}(W_{\mathrm{in}}p+\mathrm{Enc}(x))\bigr)\Bigr),\qquad T_{\beta}(p)=(1-\beta)\,p+\beta\,F_{\theta}(p;\,x), (2) where f θ f_{\theta} is a Transformer over all sites, the softmax acts per site, and Π x \Pi_{x} overwrites every site of 𝒦 \mathcal{K} with its one-hot vector. The prediction at site i i is the arg ⁡ max \arg\max over the output block of the settled p i p_{i} . Figure 1 illustrates this settling process on a Sudoku puzzle.

Since 𝒫 x \mathcal{P}_{x} is nonempty, compact, and convex and T β T_{\beta} maps it continuously into itself, a fixed point exists for every θ \theta , x x , and β ∈ ( 0 , 1 ] \beta\in(0,1] (section 2 ).

#### Why a simplex.

Existence alone does not single out the simplex: any continuous map of a compact convex set into itself has a fixed point, so a bounded latent z ∈ [ − 1 , 1 ] d z\in[-1,1]^{d} updated by tanh ⁡ ( f θ ​ ( z , x ) ) \tanh(f_{\theta}(z;x)) has one too. What the simplex adds is that every intermediate state is itself a prediction. First, the iteration can be read at any step: the arg ⁡ max \arg\max of p i p_{i} is the current answer at site i i and p i p_{i} is the model’s confidence in it, with no readout head in between. Second, the residual measures how much the prediction moves. For any two distributions | a j − b j | ≤ TV ⁡ ( a , b ) |a_{j}-b_{j}|\leq\mathrm{TV}(a,b) , so a site whose top two output probabilities differ by more than twice its per-step TV keeps its arg ⁡ max \arg\max on that step; in a latent chart, a small step in z z bounds the change of the decoded answer only through the Lipschitz constant of the head. Third, the correct solution is a known point of the state space, a vertex of 𝒫 x \mathcal{P}_{x} , so the loss pulls the state toward it directly and the distance to it can be tracked along the trajectory. Fourth, task constraints are statements about beliefs, so they can be imposed on intermediate states rather than checked after decoding (section 3.4 ).

### 3.2 Training and inference

Training unrolls the iteration from the uniform state, or with probability 0.25 0.25 from a random Dirichlet state, in two phases. A first rollout of at most D D steps runs without gradients and stops early once every puzzle in the batch has a per-site TV step below a tolerance for two consecutive steps. From its end state p ( 0 ) p^{(0)} , a tail of m m steps p ( j + 1 ) = T β ​ ( p ( j ) ) p^{(j+1)}=T_{\beta}(p^{(j)}) is differentiated. With p ~ i \tilde{p}_{i} the renormalized output block of site i i at the final state p ( m ) p^{(m)} and 𝒦 ¯ \bar{\mathcal{K}} the free sites, the loss is ℒ = 1 | 𝒦 ¯ | ∑ i ∈ 𝒦 ¯ − log p ~ i , y i + λ aux a ( p ( m ) ) + λ res 1 m ∑ j = 0 m − 1 1 n ∑ i = 1 n ∥ F ^ θ , i ( p ( j ) ; x ) − p i ( j ) ∥ 2 2 , \mathcal{L}=\frac{1}{|\bar{\mathcal{K}}|}\sum_{i\in\bar{\mathcal{K}}}-\log\tilde{p}_{i,y_{i}}+\lambda_{\mathrm{aux}}\,a(p^{(m)})+\lambda_{\mathrm{res}}\,\frac{1}{m}\sum_{j=0}^{m-1}\frac{1}{n}\sum_{i=1}^{n}\bigl\|\hat{F}_{\theta,i}(p^{(j)};x)-p^{(j)}_{i}\bigr\|_{2}^{2}, (3) where a ⁡ ( p ) a(p) is the mean auxiliary mass on free sites and F ^ θ = softmax ⁡ ( f θ ​ ( W in ​ p + Enc ⁡ ( x ) ) ) \hat{F}_{\theta}=\mathrm{softmax}\bigl(f_{\theta}(W_{\mathrm{in}}p+\mathrm{Enc}(x))\bigr) is F θ F_{\theta} before Π x \Pi_{x} . The last term is the one-step residual along the tail; because it is taken before pinning, it also asks the network to reproduce the given sites. We use λ res = 0.02 \lambda_{\mathrm{res}}=0.02 on Sudoku and λ res = 0 \lambda_{\mathrm{res}}=0 on Maze; depth distributions and other per-task settings are listed in the supplementary material.

#### Which fixed point is learned.

Existence does not say which fixed point the test-time loop reaches from the uniform start (section 2 ). Training aligns F θ F_{\theta} with that trajectory: the cross-entropy is taken at the end of a rollout from the same start, so gradients act on the states that inference actually visits rather than on equilibria it may never approach. The Dirichlet starts ask that the labeled vertex also be reached from a spread of initial beliefs, which widens its basin.

On Sudoku the rollout cap D D is redrawn every batch from a heavy-tailed distribution (Maze uses a fixed D D ), so the same weights must give the right answer after short and long rollouts. A map that passes through the solution at one depth and then moves on is penalized at the others, so the answer has to persist. Short draws also end the rollout before the state settles, so the loss sees intermediate states as well; training only at settled states would leave the map free to form confident wrong attractors in regions the loss never visits.

With λ res > 0 \lambda_{\mathrm{res}}>0 on Sudoku, the residual term additionally asks the state the model settles on to be stationary, not merely correct at the end of the tail. On Maze, λ res = 0 \lambda_{\mathrm{res}}=0 and only the final answer is supervised. Near a fixed point, where the Jacobian varies slowly along the tail, backpropagation through the m m tail steps approaches a truncated Neumann approximation of the implicit fixed-point gradient, which involves ( I − J F ) − 1 (I-J_{F})^{-1} ( Bai et al., 2019 ) .

None of this certifies convergence at test time. Training rollouts are tens of steps long (mean D ≈ 32 D\approx 32 on Sudoku), while test budgets reach 35,000 35{,}000 steps, so behavior beyond the training horizon is extrapolation; a few held-out Sudoku puzzles settle to a stable but incorrect grid.

At test time GFPR starts from the uniform state with the given sites pinned, applies p k + 1 = T β ​ ( p k ) p_{k+1}=T_{\beta}(p_{k}) for K K steps (on S 5 S_{5} , until the total-variation step falls below a tolerance; budgets in the supplementary material), and reads arg ⁡ max \arg\max on the output block of p K p_{K} . No labels, verifier, or restarts are used at test time.

### 3.3 Language models: wavefront decode