Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision — part 3/3
Cleaned-text code points [31039, 45437). Bibliography omitted here; source.md includes it.
We introduced Compute as Teacher, a framework for converting inference compute into supervision. Parallel rollouts sample the model’s uncertainty while reference estimation aggregates them into pseudo-references. A reward derivation step converts pseudo-references into RL rewards. The framework is agnostic to the aggregation method. We demonstrated synthesis as one strong option, but majority vote and best-of-N may also apply. Our core contribution is enabling reference-free RL in non-verifiable domains. Self-proposed rubrics provide stable rewards without human annotation. On HealthBench, these rubrics are similar to physician-designed ones in performance, and CaT improves over the base policy by up to 30% (relative). Trained models match or exceed inference-time aggregation quality at 9 × 9\times less test-time compute. Therefore, CaT enables spending compute once during training instead of repeatedly at inference. On verifiable domains (MATH-500), CaT performs as well as any other approach, showing the framework is ‘plug-and-play’ across domain types. Compute as Teacher provides a practical framework for post-training language models on non-verifiable and verifiable domains where reference answers are scarce, expensive, contested, or even unknown. As human annotation becomes the bottleneck for specialized domains, learning from inference compute offers one possible path forward.
## Acknowledgements
In alphabetical order, we would like to thank the following people: Joseph Brennan for assistance with rubric data, Lovish Madaan for various technical assistance, Nicola Cancedda for feedback on institutional approval processes, Stéphane Collot for guidance on rubric generation, Yonatan Gideoni for comments on a draft of this work, and Yunzhen Feng for general technical advice.
DJ was an intern at Meta at the time of this work and is supported by an AWS Studentship from the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (AIMS) (EP/S024050/1).
## Appendix A Results Tables And Additional Results
See Tables 2 , 3 , 4 , and 5 for tabular transcriptions of the results in the figures in the main body.
### A.1 RL Results Against Selection Methods
In Table 6 , we observe that CaT outperforms the selection baselines with RL, just as synthesis did at inference time because synthesis delivers a better supervision signal.
### A.2 Conditioning Rubric Generation on the Pseudo-Reference
Table 7 ablates rubric generation where conditioning on the pseudo-reference s s outperforms conditioning only on the question q q , improving results by 12% (relative). The compute spent on reference estimation improves not just the supervision target but also the quality of the derived rubrics.
### A.3 Judge Sensitivity
In Table 8 we show results on HealthBench when running with different sized judge models. We keep the evaluation benchmark judge the same. Stronger judges somewhat lead to better performance, likely due to improved rubric judgement accuracy.
### A.4 Omitting the Question in the Synthesis Aggregator
We note that in the synthesis step, we do not include the task prompt or question in the estimator’s prompt because it did not make a difference in preliminary inference-time experiments with Gemma 3 4B on MATH-500 ( + 0.004 +0.004 ). Excluding the task prompt simplifies the setup and makes no meaningful difference to performance. It also helps ensure that the aggregator will not simply act as another rollout by attempting to answer the question directly.
## Appendix B Example: Synthesis Disagrees With All Rollouts
Disagreement with all rollouts occurs across all models. The following is one among a few examples discovered with Gemma 3 4B on the MATH-500 dataset.
All rollouts failed to provide the correct answer, exhibiting calculation errors. The following is an example from the second rollout which did not compute a division correctly:
In another example, the sixth rollout made several calculation errors, inexplicably multiplying and dividing by 137 and 1 around the same place as the second rollout:
Despite this, the synthesized response identified these errors, used the correct reasoning and provided the right final response. Since the individual rollouts failed to find the correct answer, finding the right method would not be easy for the model without observing these attempts.
## Appendix C Analysis of Synthesis Outputs
To further understand how synthesis differs from raw policy rollouts, we conducted a comparison of their generated traces. Table 9 summarizes several structural characteristics of the two sets of outputs.
Synthesis outputs have similar overall length to policy rollouts, despite achieving higher accuracy. This suggests that synthesis does not simply “think longer”. Instead, it appears to leverage and reuse reasoning already present in the rollouts, synthesizing relevant fragments rather than producing entirely new derivations. Even among synthesis outputs, correct answers are significantly shorter on average ( p < 0.05 p<0.05 ), further indicating that synthesis benefits from selectively reusing already-correct partial reasoning. Interestingly, synthesis outputs contain fewer equations and substantially fewer lines than policy rollouts. Manual inspection of the traces indicates that synthesis often summarizes or enumerates intermediate possibilities that are already explored in the rollouts, rather than reconstructing these steps in detail. This behavior is consistent with synthesis acting as a reconciling mechanism over the existing reasoning traces.
We also examined stylistic markers of reasoning. Using keyword heuristics for step-by-step patterns (e.g., “first”, “second”), we observe that synthesis employs stepwise reasoning approximately 37% less often (absolute difference). Conversely, synthesis outputs contain 9.3% more verification cues (e.g., “let’s verify”, “check”), reflecting a tendency to inspect or validate reasoning rather than generate long chains.
Overall, these observations support the interpretation that synthesis primarily reconciles and synthesizes partially correct reasoning traces from the policy rollouts, rather than behaving like an additional independent rollout.
## Appendix D When Does CaT Stop Learning?
In Figure 6 , we compare the trained policy to if we apply synthesis at inference-time to the trained policy. The latter is the final teacher signal in CaT when synthesis is used as the reference estimation strategy. At this point, we note that the teacher signal is very close to the trained policy’s performance. Therefore, the model is unable to continue improving as the teacher provides very little delta to improve.
Since the synthesis step improves upon the group rollouts by resolving contradictions, synthesizing partial solutions, and inserting omissions, if it does not improve, then this indicates that the group rollouts are generally in agreement. Here, we note that the model has gone from generating diverse solutions when it was less capable to generating less diverse, but more likely solutions when it has been trained to be more capable at solving the task. This is a commonly observed issue in RL fine-tuning ( Yue et al., 2025 ; Song et al., 2025b ; Wu et al., 2025 ; Zhao et al., 2025b ) . Its presence here places a bound on the potential reference-free improvement that can be achieved via CaT.
## Appendix E GRPO Overview
Group Relative Policy Optimisation ( Shao et al., 2024 , GRPO) is a memory-efficient variant of PPO ( Schulman et al., 2017 ) that avoids a value network by using a group baseline. For each q q , we draw G G rollouts o 1 : G o_{1:G} from the policy π θ old \pi_{\theta_{\text{old}}} and optimize J GRPO ( θ ) = 𝔼 q , { o i } [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | L t ( θ ) − β D KL [ π θ ∥ π ref ] ] \begin{split}J_{\mathrm{GRPO}}(\theta)\;=\;\mathbb{E}_{q,\,\{o_{i}\}}\Bigg[&\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}L_{t}(\theta)\\ &-\;\beta\,\mathrm{D}_{\mathrm{KL}}\!\big[\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\big]\Bigg]\end{split} (6)
with the clipped surrogate L t ( θ ) = min ( r t ( θ ) A ^ i , t , clip ( r t ( θ ) , 1 − ε , 1 + ε ) A ^ i , t ) , L_{t}(\theta)=\min\!\Big(r_{t}(\theta)\,\hat{A}_{i,t},\mathrm{clip}\!\big(r_{t}(\theta),\,1-\varepsilon,\,1+\varepsilon\big)\,\hat{A}_{i,t}\Big), (7) where the importance weighting token-level ratio and the group-normalized advantage are r t ( θ ) = π θ ( o i , t ∣ q , o i , < t ) π θ old ( o i , t ∣ q , o i , < t ) , A ^ i , t = R ( q , o i ) − R ¯ G σ G . r_{t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})},\quad\hat{A}_{i,t}=\frac{R(q,o_{i})-\bar{R}_{G}}{\sigma_{G}}. (8) Here R ¯ G = 1 G ∑ j = 1 G R ( q , o j ) \bar{R}_{G}=\tfrac{1}{G}\sum_{j=1}^{G}R(q,o_{j}) is the group mean reward and σ G \sigma_{G} its standard deviation; the KL term discourages large policy drift from the reference π ref \pi_{\mathrm{ref}} (typically the initial policy π 0 \pi_{0} ).
## Appendix F Prompts
We provide two prompts for synthesis. We use the Freeform Synthesis Prompt for HealthBench questions, and the COT/Reasoning Synthesis Prompt for maths questions.
## Appendix G Example Rubrics
All examples in this section were generated from Qwen 3 4B on the HealthBench dataset.
## Appendix H Hyperparameters
We provide RL training parameters in Table 10 , SFT training parameters in Table 11 , and model sampling parameters in Table 12 . We use the verl library ( Sheng et al., 2025 ) for both RL and SFT. We also note that we apply a length penalty of − 1 -1 to responses longer than 750 750 tokens when training with HealthBench to discourage length-based reward hacking.
## Appendix I Experimental Details
##### Computing perplexity.
To compute the perplexity of the output tokens in response to a question, we calculate Perplexity ( w 1 , w 2 , … , w n | context ) = exp ( − 1 n ∑ i = 1 n log p ( w i | context , w 1 , … , w i − 1 ) ) \text{Perplexity}(w_{1},w_{2},\ldots,w_{n}|\text{context})=\exp\left(-\frac{1}{n}\sum_{i=1}^{n}\log p(w_{i}|\text{context},w_{1},\ldots,w_{i-1})\right) (9) where w 1 , w 2 , … , w n w_{1},w_{2},\ldots,w_{n} are the output tokens generated by the model. When selecting the best response for min(PPL), in practice we do not compute the exponential as minimizing entropy is the same as minimizing perplexity.
##### Computing mutual predictability.
For G = 8 G=8 rollouts we construct eight prompts, where we pick each rollout answer in turn to include last in the prompt and randomly order the other answers in the prompt before it. Then, we encode the prompt with the model and compute the token-level perplexity of the tokens in the final answer:
PPL ( a j ) = exp ( − 1 | a j | ∑ t = 1 | a j | log p ( w t ( j ) | context , a − j , w 1 ( j ) , … , w t − 1 ( j ) ) ) \text{PPL}(a_{j})=\exp\left(-\frac{1}{|a_{j}|}\sum_{t=1}^{|a_{j}|}\log p(w_{t}^{(j)}|\text{context},a_{-j},w_{1}^{(j)},\ldots,w_{t-1}^{(j)})\right) (10)
where a j a_{j} is the j j -th answer, | a j | |a_{j}| is its length in tokens, w t ( j ) w_{t}^{(j)} is the t t -th token of answer j j , and a − j a_{-j} represents the other answers included in the context. We pick the answer with the lowest perplexity as the best response:
a ∗ = arg min j ∈ { 1 , … , G } PPL ( a j ) a^{*}=\arg\min_{j\in\{1,\ldots,G\}}\text{PPL}(a_{j}) (11)
##### Supervised fine-tuning.
For our SFT experiments, we generate G = 8 G=8 rollouts with the initial policy π 0 \pi_{0} over our HealthBench training and validation splits. Then, we use the same initial policy to synthesize the rollouts per question into a synthesized estimated reference response s s . We then fine-tune the model with the estimated reference responses as targets by minimizing the cross-entropy loss
ℒ SFT = − 𝔼 ( q , s ) ∼ 𝒟 [ 1 | s | ∑ t = 1 | s | log π θ ( s t | q , s < t ) ] \mathcal{L}_{\text{SFT}}=-\mathbb{E}_{(q,s)\sim\mathcal{D}}\left[\frac{1}{|s|}\sum_{t=1}^{|s|}\log\pi_{\theta}(s_{t}|q,s_{<t})\right] (12)
where q q is the input question, s s is the estimated reference response, s t s_{t} is the t t -th token of the reference response, and 𝒟 \mathcal{D} is the training dataset. We use early stopping, using the checkpoint with the lowest validation loss to evaluate the model on the held-out 500-question HealthBench test set. We also note that we train with LoRA ( Hu et al., 2022 ) due to fast overfitting and worse results with full parameter fine-tuning.
##### RL fine-tuning.
Much of the detail for RL fine-tuning is described in the main body and other appendices. Here, we note that for math data, we extract a verifiable final answer from boxed text, e.g., boxed{...} , using regular expressions and string matching where we have instructed the model to give its final answer in this form. To extract rubric judgments and rubric generations, we instruct the model to output its answer in XML format 1 1 1 See the prompts in Appendix F and https://www.w3.org/TR/xml/ . and use a standard XML tree parser to extract the result. When RL fine-tuning with HealthBench, we use early stopping, evaluating the test set with the checkpoint that yielded the best validation score. For math, since we use the test-time reinforcement learning setting ( Zuo et al., 2025 ) , we train for a fixed number of steps.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .