Hierarchical Latent Prediction for Language Models — part 2/2

Cleaned-text code points [15073, 18659). Bibliography omitted here; source.md includes it.

Table 5 compares the four 100B-token code-data runs along three axes: total parameter count, the subset of parameters actually used during greedy NTP decoding, and training throughput. All four models share the same 1.06 B-parameter trunk and tied-free LM head, so the verifier path used at inference is identical and the additional parameters in MTP, NextLat, and HiLP are auxiliary draft components that are not required to produce the next-token distribution. NextLat adds a small ( ≈ \approx 19 M) next-latent predictor on top of the trunk, while HiLP and MTP each add a ≈ \approx 190–200 M draft module. Training throughput scales inversely with the size of the auxiliary loss graph: the NTP baseline reaches ≈ \approx 126 K tokens/s/GPU, NextLat retains ≈ \approx 83% of that throughput, HiLP retains ≈ \approx 65%, and MTP drops to ≈ \approx 49% because its four future-token heads must each be evaluated against the LM logits at every step.

## Appendix D Input-Combination Modes

We include an implementation-level ablation comparing the historical concat input-combination mode against the current glu_cross mode used by latent prediction modules. Let D D denote the hidden size, m m the latent-head hidden multiplier, and V V the vocabulary size. In the lower next-latent predictor, concat forms [ 𝐡 t ; e t + 1 ] ∈ ℝ 2 ​ D [\mathbf{h}_{t};e_{t+1}]\in\mathbb{R}^{2D} and feeds it directly to the SwiGLU predictor. This gives an intermediate width of 2 ​ m ​ D 2mD and costs 10 ​ m ​ D 2 10mD^{2} parameters and multiply-adds per token for the gate, up, and down projections, ignoring biases. By contrast, glu_cross computes W h ​ 𝐡 t ⊙ σ ⁡ ( W e ​ e t + 1 ) ∈ ℝ D , W_{h}\mathbf{h}_{t}\odot\sigma(W_{e}e_{t+1})\in\mathbb{R}^{D}, adding two D × D D\times D projections but reducing the following SwiGLU width to m ​ D mD . Its corresponding cost is therefore ( 3 ​ m + 2 ) ​ D 2 (3m+2)D^{2} . With the default m = 4 m=4 , this is 14 ​ D 2 14D^{2} versus 40 ​ D 2 40D^{2} , or about 35 % 35\% of the concat predictor cost.

The same distinction appears in the combined NTP head. For concat , the combined head maps [ 𝐡 t ; 𝐮 t ] ∈ ℝ 2 ​ D [\mathbf{h}_{t};\mathbf{u}_{t}]\in\mathbb{R}^{2D} to logits, costing 2 ​ D ​ V 2DV . For glu_cross , the gated fusion costs 2 ​ D 2 2D^{2} and the logits are produced from a D D -dimensional vector, costing D ​ V DV , for a total of D ​ V + 2 ​ D 2 DV+2D^{2} . Thus glu_cross is especially attractive when V ≫ D V\gg D : it approximately halves the combined-head logit computation while also reducing the latent predictor from a 2 ​ D 2D -wide to a D D -wide SwiGLU input.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .