Reasoning on the Simplex: Geometric Fixed-Point Models — part 3/4
Cleaned-text code points [31736, 47433). Bibliography omitted here; source.md includes it.
Recurrence in neural nets is older than the current puzzle models. Hopfield networks already iterate a state that is also the output: a binary pattern is updated until it matches a stored memory ( Hopfield, 1982 ) . Almeida–Pineda nets inject a constant input at every step and run until a fixed point ( Almeida, 1987 ; Pineda, 1987 ) ; written as h l + 1 = f ( h l , x , w ) h^{l+1}=f(h^{l},x;w) this is depth recurrence with x x clamped. Classic graph neural nets do the same on a graph, but they constrain the transition to be a contraction so the fixed point is unique, and they still decode with a separate head ( Scarselli et al., 2009 ) .
Weight-tied depth then reappeared as a way to spend test-time compute without growing the parameter count. The Universal Transformer ties one block across depth and can stop with ACT ( Dehghani et al., 2019 ; Graves, 2016 ) . Looped Transformers put a tied core between an untied prelude and coda, re-inject the input each step, and raise the loop count at test time ( Giannou et al., 2023 ; Geiping et al., 2025 ) . HRM nests two timescales and TRM collapses them to one stream; both can stop with ACT or a step cap ( Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ) . The iterate in these models is a free latent, and a head turns the last state into symbols.
Deep equilibrium models treat an infinitely deep weight-tied stack as the root of z = f θ ( z , x ) z=f_{\theta}(z;x) and differentiate at that root ( Bai et al., 2019 ) . FPRM and EqR use the same idea as the reasoning procedure itself ( Movahedi et al., 2026 ; Huang et al., 2026 ) . FPRM damps the Picard step and stops when the latent residual is small. EqR treats the map as an attractor landscape and can spend extra compute on restarts. In both, the state remains a vector in ℝ d \mathbb{R}^{d} ; a fixed point is not guaranteed by the architecture.
GFPR is an Almeida–Pineda-style depth recurrence: the input is injected every step, and the state is a field of categorical beliefs. As in a Hopfield network, the state is itself the output, but it is a learned distribution per site rather than a binary pattern matched to stored memories. The same state can carry structural constraints on a task (section 3.4 ), trained with a Fenchel–Young loss ( Blondel et al., 2020 ) .
On language, a standard Transformer has no depth recurrence: each block has its own weights and is visited once ( Radford et al., 2019 ) . Chain-of-thought spends compute by writing tokens ( Wei et al., 2022 ; Kojima et al., 2022 ) . Latent looped language models iterate a hidden state ( Geiping et al., 2025 ) . GFPR iterates a belief over the vocabulary at each site (section 3.3 , section 4.4 ).
## 6 Conclusion
We argued that a looped reasoner should iterate its prediction rather than a latent code decoded at the end. In GFPR every intermediate state is a field of categorical beliefs that can be read as an answer, its residual measures how much that answer moves, and task structure can be imposed on it as a convex set. A fixed point then exists for any parameters, as a consequence of the design rather than its purpose; which one is reached is decided by training.
On Sudoku-Extreme, Maze-Hard, and S 5 S_{5} , this state is enough to exceed the published single-trajectory numbers of latent looped reasoners at similar scale, and on the puzzle of Figure 2 GFPR settles to a fixed point where FPRM’s halt is not one. Adding task-specific convex structure helps where the relaxation is informative: the unit-flow readout raises Maze-Hard from 92.0% to 96.0%, while the Sudoku consistency readout so far only matches the product-of-simplices baseline. The language-model experiment is small, but it shows that the same update works when each site is a distribution over a vocabulary: after 24 Picard steps a 201M model trained on 6B tokens is competitive with GPT-2 small and medium on zero-shot multiple choice, and that quality appears only after iteration.
The evidence has clear limits. The baselines are quoted rather than re-run under a matched training recipe and inference budget, so the share of the gain due to the state space is not isolated. The language-model comparison is not compute-matched and scores likelihoods rather than generated text. A few Sudoku puzzles settle to a stable but wrong grid; because the state can be checked against the constraints without labels, restarts selected by constraint violation are a natural next step.
## Appendix A FPRM and GFPR side by side
Figure 3 is one step of each method. The first three rows are the loop. The last row is not a further step. It is the reason a fixed point does or does not come for free.
FPRM can stop when the step is small; GFPR runs for a fixed step budget at test time. They do not certify the same thing. In FPRM the step is motion of y y . A head still turns y y into digits; contraction is sufficient for a fixed point of F F and for local convergence, but neither is guaranteed by the architecture. In GFPR the step is motion of p p , and p p is already the prediction. Clues stay pinned, the state stays in a compact product of simplices, so Brouwer gives a fixed point for every θ \theta ; which equilibrium the iteration reaches is still a training question.
## Appendix B Model and training settings
Table 3 lists the settings of the two released GFPR runs. Both use the same Transformer trunk over all sites. Inside one application of F θ F_{\theta} the trunk is run for a few self-conditioning passes: each pass takes the previous pass’s softmax output as extra input, and the last one is the output of F θ F_{\theta} . Before each softmax the logits are soft-capped as 15 tanh ( ℓ / 15 ) 15\tanh(\ell/15) .
The training rollout (Section 3.2 of the main text) has two phases. The first runs without gradients for at most D D steps (random on Sudoku, fixed on Maze) and stops early once every puzzle in the batch has kept its maximum per-site TV step below a tolerance for two consecutive steps. On Sudoku, D D is drawn per batch as D = 1 + Poisson ( e τ ) D=1+\mathrm{Poisson}(e^{\tau}) with τ ∼ 𝒩 ( log D ¯ − σ 2 / 2 , σ 2 ) \tau\sim\mathcal{N}(\log\bar{D}-\sigma^{2}/2,\ \sigma^{2}) , which has mean close to D ¯ \bar{D} and a heavy right tail; on Maze-Hard, D D is fixed. The second phase differentiates a tail of m m steps, and the loss is applied to its final state. With probability 0.25 0.25 a rollout starts from a random Dirichlet state instead of the uniform one. Evaluation and checkpoint selection use an exponential moving average of the weights.
## Appendix C Evaluation protocol
#### Sudoku-Extreme.
Checkpoints are selected on a stratified 1,000-puzzle validation sample of the test_hard split of sapientinc/sudoku-extreme-1k ; the remaining test_hard puzzles form a disjoint local test split for development. The selected checkpoint reaches 94.0% exact match on a 500-puzzle subsample of the validation slice with a 20,000-step budget. Headline evaluation is computed on all 422,786 puzzles of sapientinc/sudoku-extreme ( test.csv )—a separate Hugging Face release from sudoku-extreme-1k —with the same cap as the released FPRM Sudoku-Extreme evaluation: a single trajectory from the uniform state, β = 0.7 \beta=0.7 , and at most 35,000 damped Picard steps (early stop when every site’s total-variation step falls below 5 ⋅ 10 − 3 5\cdot 10^{-3} for two consecutive steps); that file is never used for checkpoint selection. Exact match is 95.11% and cell accuracy 96.93%.
#### Maze-Hard.
Evaluation uses all 1,000 test mazes of sapientinc/maze-30x30-hard-1k , exactly 1,024 damped Picard steps, and β = 0.3 \beta=0.3 . Exact match is 92.0%, cell accuracy 99.67%, and 99.7% of settled parent fields trace a connected route from goal to start.
#### S5.
Training uses 2,000,000 sequences of 32 group updates, and evaluation uses all 1,000 test sequences of length 128. Inference runs at most 256 damped Picard steps at β = 0.7 \beta=0.7 from the uniform state and stops early when the maximum per-site total variation falls below 5 ⋅ 10 − 3 5\cdot 10^{-3} . On the selected checkpoint every test sequence is correct at every prefix length, so both sequence accuracy and final-state accuracy are 100%. The mean number of steps to that tolerance is 81.
#### Parent-field target.
A breadth-first tree defines a unique parent only once the order in which neighbors are expanded is fixed, and most Maze-Hard instances have several shortest routes. We recover the expansion order of the dataset generator from the data and check that it reproduces every labeled route, so the traced route is compared with the same route that the published numbers score.
## Appendix D Task-specific convex sets
This section gives the convex sets used by the constrained models in section 3.4 .
#### Sudoku.
For the Sudoku-Extreme dataset, let q r c d q_{rcd} denote the coordinate associated with digit d ∈ { 1 , … , 9 } d\in\{1,\ldots,9\} at cell ( r , c ) (r,c) , and let ℬ \mathcal{B} denote the collection of nine 3 × 3 3\times 3 boxes. For a puzzle x x , let d r c d_{rc} denote the given digit at each clue cell ( r , c ) (r,c) . We define 𝒞 Sud ( x ) = { q ∈ [ 0 , 1 ] 9 × 9 × 9 : ∑ d = 1 9 q r c d = 1 ∀ r , c , ∑ c = 1 9 q r c d = 1 ∀ r , d , ∑ r = 1 9 q r c d = 1 ∀ c , d , ∑ ( r , c ) ∈ B q r c d = 1 ∀ B ∈ ℬ , d , q r c , d r c = 1 for every clue cell ( r , c ) . } . \mathcal{C}_{\mathrm{Sud}}(x)=\left\{q\in[0,1]^{9\times 9\times 9}:\begin{array}[]{ll}\displaystyle\sum_{d=1}^{9}q_{rcd}=1&\forall r,c,\\[2.0pt] \displaystyle\sum_{c=1}^{9}q_{rcd}=1&\forall r,d,\\[2.0pt] \displaystyle\sum_{r=1}^{9}q_{rcd}=1&\forall c,d,\\[2.0pt] \displaystyle\sum_{(r,c)\in B}q_{rcd}=1&\forall B\in\mathcal{B},d,\\[2.0pt] q_{rc,d_{rc}}=1&\text{for every clue cell }(r,c).\end{array}\right\}. (9) The first family of equalities makes ( q r c 1 , … , q r c 9 ) (q_{rc1},\ldots,q_{rc9}) a probability vector for every cell. The remaining equalities require each digit to have total coordinate sum one in every row, column, and box, while the final constraints fix the clue coordinates. Every valid completed Sudoku grid corresponds to an integral point of 𝒞 Sud ( x ) \mathcal{C}_{\mathrm{Sud}}(x) . The converse does not hold for arbitrary feasible points because the polytope can contain fractional solutions; consequently, feasibility in 𝒞 Sud ( x ) \mathcal{C}_{\mathrm{Sud}}(x) does not by itself imply a valid discrete Sudoku completion.
We use the power regularizer Ω α \Omega_{\alpha} defined in section 3.4 and define q ∗ ( z ) = arg max q ∈ 𝒞 Sud ( x ) { z ⊤ q − Ω α ( q ) } . q^{*}(z)=\operatorname*{arg\,max}_{q\in\mathcal{C}_{\mathrm{Sud}}(x)}\left\{z^{\top}q-\Omega_{\alpha}(q)\right\}. (10) We evaluate Eq. equation 10 through its convex dual. After fixing the clue coordinates, let F F denote the remaining coordinates and write the equality constraints as A F q F = b ~ x A_{F}q_{F}=\widetilde{b}_{x} . For a dual variable λ \lambda , define s = z F − A F ⊤ λ s=z_{F}-A_{F}^{\top}\lambda . Maximization of the Lagrangian with respect to the primal coordinates yields q i ( λ ) = min { 1 , [ ( α − 1 ) [ s i ] + ] 1 / ( α − 1 ) } . q_{i}(\lambda)=\min\left\{1,\bigl[(\alpha-1)[s_{i}]_{+}\bigr]^{1/(\alpha-1)}\right\}. (11) The dual problem is solved numerically, and the resulting primal point is recovered from Eq. equation 11 . Numerical feasibility is measured by the equality residual ‖ A F q F ( λ ) − b ~ x ‖ \|A_{F}q_{F}(\lambda)-\widetilde{b}_{x}\| .
The choice of α \alpha has only a modest effect on the number of recurrent iterations needed to reach the final accuracy: it shifts the transition point slightly, while the curves nearly coincide after sufficient iteration. The S5 and Sudoku ablations are reported in figure 4 and Table 4 .
#### Maze.
For Maze-Hard dataset, let G x = ( V x , E x ) G_{x}=(V_{x},E_{x}) be the directed graph whose vertices are the open cells and whose edge set contains both orientations of every adjacent pair of open cells. Let B x ∈ ℝ | V x | × | E x | B_{x}\in\mathbb{R}^{|V_{x}|\times|E_{x}|} be its node–edge incidence matrix, with coefficient + 1 +1 for an outgoing edge and − 1 -1 for an incoming edge. To match the parent-pointer decoding direction, we orient the flow from the goal g g to the start s s and define b x ∈ ℝ | V x | b_{x}\in\mathbb{R}^{|V_{x}|} by ( b x ) v = { 1 , v = g , − 1 , v = s , 0 , otherwise . (b_{x})_{v}=\begin{cases}1,&v=g,\\ -1,&v=s,\\ 0,&\text{otherwise}.\end{cases} The flow polytope 𝒞 flow ( x ) \mathcal{C}_{\mathrm{flow}}(x) is defined in equation 7 . The equality B x f = b x B_{x}f=b_{x} imposes flow conservation at every vertex other than g g and s s , together with net outflow one at g g and net inflow one at s s . The continuous relaxation may contain fractional flows and nonzero circulations. Therefore, membership in 𝒞 flow ( x ) \mathcal{C}_{\mathrm{flow}}(x) does not imply that f f is the incidence vector of a simple path, nor does it imply shortestness or uniqueness. These properties are evaluated on the decoded discrete path separately.
#### Fenchel–Young training.
Training uses the Fenchel–Young loss equation 8 and its gradient ∇ z L Ω , 𝒞 ( z , y ) = Ψ 𝒞 ( x ) ( z ) − y \nabla_{z}L_{\Omega,\mathcal{C}}(z,y)=\Psi_{\mathcal{C}(x)}(z)-y when the argmax in equation 5 is unique (Danskin’s theorem). The structured map can be applied either at every recurrent step or only at the terminal readout (section 3.4 lists which construction each benchmark uses). In the recurrent construction, the structured component of every iterate belongs to 𝒞 ( x ) \mathcal{C}(x) , and differentiation through an unrolled recurrence includes derivatives of the recurrent applications of Ψ 𝒞 ( x ) \Psi_{\mathcal{C}(x)} . In the terminal-readout construction, the recurrence remains in the product-simplex state space of section 3.1 , and Ψ 𝒞 ( x ) \Psi_{\mathcal{C}(x)} is applied only to the final score vector, so the gradient with respect to the final scores does not differentiate through the numerical iterations used to compute the structured prediction.
## Appendix E Simplex language-model decode
Section 3.3 is implemented for a byte-pair TinyStories model in talgat/lm/ . The trunk is a causal Transformer with weights tied across Picard steps. Training is the relax-plus-tail unroll of section 3.2 : a non-differentiated settle, then a differentiated tail, with next-token cross-entropy on the output block of p t p_{t} against x t + 1 x_{t+1} . Open-ended generation uses wavefront decode ( generate_progressive in talgat/lm/code/decode_lm.py ). The defaults are soft-ahead width W = 8 W=8 , damping β = 0.5 \beta=0.5 , sharpening temperature τ = 1 / 16 \tau=1/16 , and a commit when either the top-two margin or a total-variation stability test fires, after a minimum number of Picard steps since the previous commit. There is no key–value cache: every Picard step is a full forward over the active length, so iteration counts describe this decoder rather than a cached autoregressive Transformer. The FineWeb-Edu multiple-choice scores in section 4.4 use fixed- K K teacher-forced unrolling, not wavefront sampling.
## Appendix F FineWeb-Edu language-model evaluation
The 201M run in section 4.4 is trained on 6,000,148,480 FineWeb-Edu tokens for 36,622 steps. The trunk has width 2048, 16 heads, and two weight-tied blocks; the vocabulary has 16,384 byte-pair tokens and 16 auxiliary coordinates; the context is 512. Training uses β = 0.5 \beta=0.5 , 16 no-gradient steps and an 8-step differentiated tail, and Muon with AdamW. Reported scores use EMA weights.