Reasoning on the Simplex: Geometric Fixed-Point Models — part 2/4
Cleaned-text code points [15871, 31736). Bibliography omitted here; source.md includes it.
The same state applies to a token sequence. Site t t holds a belief p t p_{t} over the vocabulary, and training fits the output block of p t p_{t} to the next token x t + 1 x_{t+1} , with the embedding of the true x t x_{t} as conditioning. One Picard step is one causal pass of f θ f_{\theta} . On Sudoku and mazes every free site is updated for the full inference depth and the whole answer is read at the end (section 3.2 ). Under causal attention the update at site t t depends only on sites ≤ t \leq t , so p t p_{t} is a prediction of x t + 1 x_{t+1} only after x 1 , … , x t x_{1},\ldots,x_{t} are fixed. Open-ended generation therefore alternates Picard steps with commits, and we use wavefront decode for it. Wavefront decode is purely an inference procedure: the loss equation 3 is teacher-forced, so F θ F_{\theta} is never trained on an expected embedding. The multiple-choice scores of section 4.4 need no generation and are computed from teacher-forced beliefs.
Wavefront decode writes the sequence left to right. It tracks a frontier f f : the position of the next token that is not yet fixed. When the belief p f − 1 p_{f-1} at the site before the frontier looks ready, we commit x f = arg max p f − 1 x_{f}=\arg\max p_{f-1} , pin that site to a one-hot vector, and move f f forward by one. While we wait for readiness, we do not freeze the entire suffix: up to W W sites ahead of f f (the soft-ahead width) keep taking Picard steps so later beliefs can start to form before the prefix is complete.
Embeddings follow the same split. Sites that are already committed feed the network their true token vectors. Inside the window, site i i conditions on a soft embedding built from the belief at i − 1 i-1 , e i = ∑ w q i − 1 , w Emb ( w ) , q i − 1 = softmax ( log p i − 1 / τ ) , e_{i}=\sum_{w}q_{i-1,w}\,\mathrm{Emb}(w),\qquad q_{i-1}=\mathrm{softmax}(\log p_{i-1}/\tau), (4) where q i − 1 q_{i-1} is a temperature-sharpened version of p i − 1 p_{i-1} . Sites beyond the window are not updated until the frontier moves and they enter the window. On the very first segment, the given prompt is run with teacher-forced embeddings only, so the readiness test at f f is not applied to beliefs that are still essentially uniform from the cold start.
Readiness of p f − 1 p_{f-1} is a gap between its top two probabilities, a total-variation residual that has stayed small for several steps, or the per-token step cap. A minimum number of steps is required after every commit. The first steps after a commit are small even when the state is far from equilibrium, and a raw residual test treats that pause as convergence. Pinning p f − 1 p_{f-1} to the committed one-hot makes the soft embedding at the next site match the ordinary token embedding of x f x_{f} .
Because the update at site t t ignores sites > t >t , prefix order is the order the dependencies allow. The soft window lets up to W W later beliefs move before their prefix is frozen. A margin test can commit a sharp belief that is still moving; a residual test can wait out a belief that has already converged to a flat distribution. Default W W , tolerances, and the implementation are in section E .
### 3.4 Task-specific convex constraints
The product-of-simplices state does not couple sites. We encode global task structure by a compact convex relaxation 𝒞 ( x ) \mathcal{C}(x) : a terminal structured readout for Sudoku and Maze-Hard, and the recurrent state for S 5 S_{5} . Let q q denote the structured answer variable, distinct from the state p p of section 3.1 . For nonempty compact convex 𝒞 ( x ) ⊂ ℝ d \mathcal{C}(x)\subset\mathbb{R}^{d} and a continuous convex regularizer Ω \Omega , define Φ x ( z ) \displaystyle\Phi_{x}(z) = max q ∈ 𝒞 ( x ) { ⟨ z , q ⟩ − Ω ( q ) } , \displaystyle=\max_{q\in\mathcal{C}(x)}\left\{\langle z,q\rangle-\Omega(q)\right\}, (5) Ψ 𝒞 ( x ) ( z ) \displaystyle\Psi_{\mathcal{C}(x)}(z) = arg max q ∈ 𝒞 ( x ) { ⟨ z , q ⟩ − Ω ( q ) } . \displaystyle=\operatorname*{arg\,max}_{q\in\mathcal{C}(x)}\left\{\langle z,q\rangle-\Omega(q)\right\}. Here Ψ 𝒞 ( x ) \Psi_{\mathcal{C}(x)} is a structured analogue of softmax: it maps unconstrained scores to an answer state satisfying the convex task constraints. If the maximizer is unique, compactness and continuity make Ψ 𝒞 ( x ) \Psi_{\mathcal{C}(x)} continuous in z z .
When the structured variable q q is used as the recurrent state, we update it as T β , 𝒞 ( q , x ) = ( 1 − β ) q + β Ψ 𝒞 ( x ) ( f θ ( q , x ) ) , β ∈ ( 0 , 1 ] . T_{\beta,\mathcal{C}}(q;x)=(1-\beta)q+\beta\,\Psi_{\mathcal{C}(x)}\bigl(f_{\theta}(q,x)\bigr),\qquad\beta\in(0,1]. (6) Both terms lie in 𝒞 ( x ) \mathcal{C}(x) , so convexity keeps every update in 𝒞 ( x ) \mathcal{C}(x) . Thus T β , 𝒞 T_{\beta,\mathcal{C}} is a continuous self-map, and the fixed-point existence argument of section 2 applies.
#### Instantiations.
For Sudoku, q r c d q_{rcd} is the belief that cell ( r , c ) (r,c) takes digit d d . The relaxation 𝒞 Sud ( x ) \mathcal{C}_{\mathrm{Sud}}(x) requires a distribution on each cell, unit total mass of each digit in every row, column, and 3 × 3 3\times 3 box, and q r c , d r c = 1 q_{rc,d_{rc}}=1 on clues (equation 9 ). Its points may be fractional, so membership does not itself certify a valid grid. We use the power regularizer Ω α ( q ) = 1 α ( α − 1 ) ∑ i q i α \Omega_{\alpha}(q)=\frac{1}{\alpha(\alpha-1)}\sum_{i}q_{i}^{\alpha} with α > 1 \alpha>1 and evaluate Ψ 𝒞 Sud ( x ) \Psi_{\mathcal{C}_{\mathrm{Sud}}(x)} through its dual (section D ).
For Maze-Hard, parent pointers induce a unit flow from the goal to the start on directed grid edges. With incidence matrix B x B_{x} and right-hand side b x b_{x} (source at the goal, sink at the start), 𝒞 flow ( x ) = { f ∈ [ 0 , 1 ] | E x | : B x f = b x } . \mathcal{C}_{\mathrm{flow}}(x)=\left\{f\in[0,1]^{|E_{x}|}:B_{x}f=b_{x}\right\}. (7) Flow conservation does not enforce a simple or shortest path; we decode the resulting parent field and evaluate the route separately.
For S 5 S_{5} , the structured state is a doubly stochastic matrix X ∈ ℬ 5 X\in\mathcal{B}_{5} , parameterized as X = ∑ π ∈ S 5 w π P π X=\sum_{\pi\in S_{5}}w_{\pi}P_{\pi} over the 120 permutation matrices P π P_{\pi} . The weights w π w_{\pi} are obtained by applying either softmax or α \alpha -entmax ( Peters et al., 2019 ) to the permutation scores.
#### Fenchel–Young readout.
For a target y ∈ 𝒞 ( x ) y\in\mathcal{C}(x) we use the Fenchel–Young loss ( Blondel et al., 2020 ) L Ω , 𝒞 ( z , y ) = Φ x ( z ) + Ω ( y ) − ⟨ z , y ⟩ ; L_{\Omega,\mathcal{C}}(z,y)=\Phi_{x}(z)+\Omega(y)-\langle z,y\rangle; (8) when the maximizer in equation 5 is unique, ∇ z L Ω , 𝒞 ( z , y ) = Ψ 𝒞 ( x ) ( z ) − y \nabla_{z}L_{\Omega,\mathcal{C}}(z,y)=\Psi_{\mathcal{C}(x)}(z)-y . In Table 1 , Sudoku and Maze retain the original recurrent state p p , including its auxiliary coordinates, and apply Ψ 𝒞 ( x ) \Psi_{\mathcal{C}(x)} only to the final scores. Thus the structured solver is a terminal readout rather than part of the recurrent loop. The constrained S 5 S_{5} model instead uses the doubly stochastic matrix X X as its recurrent state and applies the structured update at every step. Solver details and the entmax α \alpha ablation are in section D .
## 4 Experiments
### 4.1 Tasks
We evaluate on three structured benchmarks used by recent looped reasoners ( Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ; Movahedi et al., 2026 ) , all scored by exact match of the full answer, and on zero-shot language-model multiple choice.
#### Sudoku-Extreme.
9 × 9 9\times 9 puzzles with unique solutions, selected to be hard ( Wang et al., 2025 ) . Training uses the 1,000-puzzle split with validity-preserving augmentations (digit relabeling, band and stack permutations, transposition). Checkpoints are chosen on a stratified 1,000-puzzle validation slice of the test_hard file in sapientinc/sudoku-extreme-1k ; the headline number is on the separate public file sapientinc/sudoku-extreme test.csv ( 422,786 422{,}786 puzzles) used by HRM and FPRM, which is not used for model selection (section C ). Headline evaluation matches the released FPRM Sudoku-Extreme setup ( Movahedi et al., 2026 ) : one trajectory from the uniform state, damping β = 0.7 \beta=0.7 , and at most 35,000 35{,}000 damped Picard steps (the same step cap as FPRM on this benchmark). Each cell is a site with K = 9 K=9 digits, and the clues are pinned.
#### Maze-Hard.
30 × 30 30\times 30 mazes whose shortest path is longer than 110 steps, with 1,000 training and 1,000 test mazes ( Wang et al., 2025 ) . Rather than predicting the route mask, GFPR predicts at every open cell the direction to its parent in a breadth-first tree rooted at the start; the route is recovered by following parent pointers from the goal and compared with the labeled route. We train with the eight dihedral symmetries of the square, which the parent field respects because it is recomputed from the transformed maze. We evaluate on all 1,000 test mazes, as in prior work on this benchmark.
#### S 5 S_{5} state tracking.
Each instance is an initial arrangement of five elements followed by a sequence of permutations from the symmetric group S 5 S_{5} , and the model must output the final arrangement ( Merrill et al., 2024 ) . Following Movahedi et al. (2026) , models are trained on sequences of up to 32 updates and evaluated on up to 128, which tests length generalization. The reported quantity is exact accuracy of the final arrangement. We enumerate the 120 120 permutations in lexicographic order and represent each input permutation and each target prefix product by its index. GFPR encodes every step as a site on the product of simplices over these group elements, with no coupling across sites, and scores a sequence correct only when every prefix is correct. The constrained variant keeps a doubly stochastic state X = ∑ π w π P π X=\sum_{\pi}w_{\pi}P_{\pi} , where P π P_{\pi} are the 120 permutation matrices and the vertex weights w w are a softmax of the scores ⟨ Z , P π ⟩ \langle Z,P_{\pi}\rangle ; the entmax variant replaces this softmax by α \alpha -entmax.
#### Language modeling.
The same product-of-simplices state is trained as a causal language model on 6B tokens of FineWeb-Edu ( Penedo et al., 2024 ) (section 3.3 ). The released run has 201M parameters (width 2048, 16 heads, two weight-tied blocks, vocabulary 16,384 plus 16 auxiliary coordinates, context 512). We score four zero-shot multiple-choice tasks with next-token log-likelihood, not generated text: ARC-Easy ( Clark et al., 2018 ) , SciQ ( Welbl et al., 2017 ) , PIQA ( Bisk et al., 2020 ) , and HellaSwag ( Zellers et al., 2019 ) . GPT-2 small (124M) and GPT-2 medium (355M) ( Radford et al., 2019 ) are the comparison points.
### 4.2 Baselines
We compare with looped reasoners at a similar parameter scale: HRM ( Wang et al., 2025 ) , TRM ( Jolicoeur-Martineau, 2025 ) , EqR ( Huang et al., 2026 ) , and FPRM ( Movahedi et al., 2026 ) . For Sudoku-Extreme and Maze-Hard, every model in Table 1 is reported on the same public test files: sapientinc/sudoku-extreme test.csv ( N = 422,786 N{=}422{,}786 ) and sapientinc/maze-30x30-hard-1k test.csv ( N = 1,000 N{=}1{,}000 ), as in HRM/TRM/FPRM ( Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ; Movahedi et al., 2026 ) . We do not re-run the baselines; quoted accuracies come from Table 1 of Movahedi et al. (2026) , with TRM-MLP from Jolicoeur-Martineau (2025) . Fair comparison therefore requires matching not only the test puzzles but also the inference budget (unroll steps and damping; for FPRM, also residual tolerance and step-size decay). On Maze and S 5 S_{5} our released runs use 1,024 1{,}024 Picard steps at β = 0.3 \beta=0.3 and up to 256 256 at β = 0.7 \beta=0.7 , respectively (full protocol in section C ). The training recipes also differ (for example, GFPR uses dihedral augmentation on Maze-Hard and FPRM does not), so Table 1 compares published systems and does not by itself isolate the effect of the state space. Unless noted, every method decodes a single trajectory.
### 4.3 Results
Table 1 summarizes the results. On Sudoku-Extreme, GFPR solves 95.1% of the 422,786 test puzzles (96.9% of cells) from a single trajectory, above FPRM (94.2%) and the other single-trajectory models at 5–7M parameters. EqR reaches 99.8% when it runs 128 restarts per puzzle and selects by residual ( Huang et al., 2026 ) ; this breadth scaling is orthogonal to GFPR and could be combined with it. On Maze-Hard, GFPR solves 92.0% of the 1,000 test mazes, compared with 87.0% for FPRM. FPRM trains on Maze-Hard without augmentation, whereas GFPR uses dihedral augmentation, as TRM does. On S 5 S_{5} , trained on sequences of at most 32 updates and tested at length 128 with 320 effective layers, FPRM reaches 98.8% and plain TRM 39.4%; a causal convolution, which is not part of the original TRM, raises TRM to 97.2% (means over seeds, ± 0.9 \pm 0.9 , ± 1.9 \pm 1.9 , ± 2.5 \pm 2.5 ; Movahedi et al., 2026 ). GFPR, a 5.6M product-of-simplices model and a single seed, reaches 100.0% sequence accuracy on the 1,000 test sequences, which implies final-state accuracy. The constrained models of section 3.4 help most on Maze-Hard, where the unit-flow polytope raises exact match from 92.0% to 96.0%. On S 5 S_{5} both the softmax and the α \alpha -entmax variants reach 100.0%. On Sudoku, with α = 1.75 \alpha=1.75 , the consistency polytope scores 94.99% against 94.95% for plain GFPR on the same 23,680 random test puzzles, so it matches but does not yet improve on the product of simplices.
#### What the belief state shows.
Because the state is the prediction, its failures can be inspected without a decoder. On Maze-Hard, 99.67% of cells carry the correct parent pointer although only 92.0% of mazes are solved exactly, and 99.7% of settled parent fields trace a connected route from goal to start: most failures are a valid route that differs from the labeled one, not a broken field. On Sudoku the gap is similar, with 96.9% of cells correct against 95.1% of grids. The residual also serves as a stopping rule: on S 5 S_{5} at length 128, stopping once the largest per-site total-variation step falls below 5 ⋅ 10 − 3 5\cdot 10^{-3} takes 81 steps on average against a cap of 256, and every test sequence is still correct.
### 4.4 Language-model results
Table 2 compares the FineWeb-Edu model with GPT-2. One Picard step is a weak language model on every task. Twenty-four steps, the depth used in training, raise ARC-Easy from 32.0% to 51.9%, SciQ from 37.4% to 74.4%, PIQA from 55.6% to 63.1%, and HellaSwag from 27.8% to 33.2%. The gain is not gradual: up to eight steps the scores stay at or below the one-step level, and most of the improvement arrives between K = 8 K{=}8 and K = 16 K{=}16 . Training settles for 16 steps before its differentiated tail, and the beliefs become useful at about that depth. At K = 24 K=24 GFPR-LM is above GPT-2 small on all four tasks and above GPT-2 medium on ARC-Easy; it remains below GPT-2 medium on SciQ, PIQA, and HellaSwag. The comparison is not compute-matched: GPT-2 pays one trunk pass per token, while each Picard step is a full pass over the prompt and continuation. The scores are likelihoods of answer choices, not samples from the wavefront decoder of section 3.3 . Held-out FineWeb-Edu validation perplexity is 18.9. The experiment is small by current standards: one 201M model, 6B tokens, and four likelihood-based benchmarks. We read it as a proof of concept rather than a competitive language model: the same belief-state update, with the output simplex enlarged to the vocabulary, trains into a working language model whose quality comes from iteration (protocol in section F ).
## 5 Related Work