Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision — part 2/3
Cleaned-text code points [15633, 31039). Bibliography omitted here; source.md includes it.
When correctness is programmatically checkable, we reward agreement with the pseudo-reference: R ver ( o ; s ) = 𝟏 [ answer ( o ) = answer ( s ) ] R_{\text{ver}}(o;s)=\mathbf{1}[\texttt{answer}(o)=\texttt{answer}(s)] (2) For math problems, answer ( ⋅ ) \texttt{answer}(\cdot) extracts the final boxed expression and we check string equivalence. This is intentionally simple as the complexity in verifiable domains lies in reference estimation rather than reward derivation.
#### 3.2.2 Non-Verifiable Domains
Non-verifiable domains lack such programmatic checkers because there are numerous ways to express equivalently strong answers and therefore rule-based methods are impractical. The naive alternative, direct LLM-as-judge scoring (e.g., “rate this response 1–10”), suffers from well-documented issues such as inconsistency across prompts, bias toward verbose or stylistically fluent responses, and general susceptibility to reward hacking ( Zheng et al., 2023 ; Arora et al., 2025 ) .
We instead decompose judgment into self-proposed rubrics : binary criteria generated from the pseudo-reference and scored independently (Figure 2 ).
##### Rubric Generation.
The anchor generates a rubric ℛ = { r 1 , … , r n } \mathcal{R}=\{r_{1},\ldots,r_{n}\} from the pseudo-reference: ℛ ∼ π 0 ( ⋅ ∣ p rub , s ) \mathcal{R}\sim\pi_{0}\!\left(\cdot\mid p_{\text{rub}},s\right) (3) Each criterion r j r_{j} is a binary, verifiable statement capturing an important property of s s . We prompt for n ≥ 5 n\geq 5 specific criteria. The full rubric generation prompt p rub p_{\text{rub}} is provided in Appendix F .
##### Rubric Scoring.
An independent judge model π J \pi_{J} evaluates whether a rollout o o satisfies each criterion: R rub ( o ; ℛ ) = 1 n ∑ j = 1 n [ π J ( o , r j ) = “yes” ] R_{\text{rub}}(o;\mathcal{R})=\frac{1}{n}\sum_{j=1}^{n}\mathbf{1}\!\left[\pi_{J}(o,r_{j})=\text{``yes''}\right] (4) The reward is the fraction of criteria satisfied according to the judge π J \pi_{J} .
##### Why Rubrics?
Decomposition offers three advantages: 1. Reliability. Each binary check is simple enough for an LLM to answer consistently. “Does this response recommend professional consultation?” is easier to judge than “Is this response good?”
2. Auditability. One can inspect which criteria a rollout satisfied or failed, enabling debugging and analysis. Direct judge scores are opaque.
3. Reduced surface-form bias. A response can satisfy “recommends lifestyle modifications” regardless of exact phrasing, length, or formatting. Therefore, rubrics help to reward content rather than style.
##### Example.
Consider a user asking about managing mild hypertension. Synthesis produces a pseudo-reference recommending dietary changes, exercise, and medical follow-up. The anchor generates rubrics: 1. Acknowledges the user’s stated symptoms
2. Suggests lifestyle modifications (diet, exercise)
3. Recommends consulting a healthcare provider
4. Avoids providing a definitive diagnosis
5. Uses empathetic and appropriate tone
A rollout satisfying criteria 1, 2, 3, and 5 but failing criterion 4 receives reward R = 4 / 5 = 0.8 R=4/5=0.8 .
### 3.3 Training
We instantiate CaT with Group Relative Policy Optimization ( Shao et al., 2024 , GRPO;) , though the reward formulation is compatible with any policy gradient method that samples multiple rollouts per prompt.
##### GRPO Overview.
For each prompt q q , GRPO samples G G rollouts from the current policy and computes advantages relative to the group mean: A ^ i = R ( o i , s ) − R ¯ G σ G \hat{A}_{i}=\frac{R(o_{i};s)-\bar{R}_{G}}{\sigma_{G}} (5) where R ¯ G = 1 G ∑ j = 1 G R ( o j , s ) \bar{R}_{G}=\frac{1}{G}\sum_{j=1}^{G}R(o_{j};s) is the mean reward and σ G \sigma_{G} its standard deviation. The policy is updated via a clipped surrogate objective with KL regularization to the initial policy π 0 \pi_{0} . Full details are in Appendix E .
##### Integration with CaT.
GRPO already generates G G parallel rollouts for advantage estimation. CaT reuses these rollouts for reference estimation, adding only: 1. One synthesis call to generate s s from o 1 : G o_{1:G}
2. For non-verifiable domains: one rubric generation call, then n × G n\times G judge calls to score n n criteria
The synthesis call is comparable in cost to generating one additional rollout. Rubric scoring requires n × G n\times G judge calls per prompt, which we parallelize. These calls are low in cost as they require very few output tokens (yes/no). In practice, reference estimation and reward derivation add modest overhead relative to the cost of generating G G rollouts.
##### Training Loop.
Algorithm 1 summarizes one iteration of CaT. For each prompt: (1) sample rollouts from the current policy, (2) estimate a pseudo-reference, (3) derive rewards (programmatic for verifiable, rubric-based for non-verifiable), and (4) update the policy via GRPO.
##### Inference-Time Use.
Reference estimation can also be used purely at inference time, without RL training. Given a prompt, generate G G rollouts from the base policy π 0 \pi_{0} , estimate a pseudo-reference, and return s s as the final response. This provides immediate accuracy gains at the cost of ≥ G × \geq G\times inference compute. However, the gains do not transfer to the model weights as each deployment pays the full cost.
CaT amortizes these gains as after training, the policy π t \pi_{t} produces responses of similar or better quality than inference-time synthesis while using only a single output. In our experiments, CaT matches or exceeds inference-time synthesis at 9 × 9\times less test-time compute.
## 4 Experiments
We evaluate two aspects of our contribution. Mainly, CaT as a training framework, where we use synthesis for reference estimation. Secondarily, we also evaluate synthesis alone, i.e., reference estimation without training, to isolate the gains from each component. Our experiments address three key questions:
1. Does CaT improve over the base policy, and can it match or exceed inference-time synthesis at lower test-time cost?
2. Do self-proposed rubrics provide effective rewards in non-verifiable domains?
3. How does synthesis compare to selection baselines for reference estimation?
### 4.1 Setup
##### Datasets.
We evaluate on two domains: • HealthBench ( Arora et al., 2025 ) : 5,000 freeform healthcare conversations between physicians and users. This is a non-verifiable domain where responses are qualitative, multiple valid answers exist, and no programmatic checker is available. We hold out 500 conversations with physician-designed evaluation rubrics for testing; the remainder is used for reference-free training and validation.
• MATH-500 ( Hendrycks et al., 2021 ) : 500 competition mathematics problems. This is a verifiable domain in which final answers can be checked programmatically. Following Zuo et al. (2025) , we train and test on the same 500 problems, where training has no access to any reference labels, as a test-time training setup.
HealthBench is our primary testbed because non-verifiable domains represent the harder, unsolved problem. MATH-500 serves a different purpose, demonstrating that CaT generalizes across domain types without modification, functioning as a unified ‘plug-and-play’ framework rather than a method specific to only non-verifiable domains.
##### Models.
We evaluate three instruction-tuned model families at the 4–8B scale: Gemma 3 4B Instruct ( Kamath et al., 2025 ) , Qwen 3 4B Instruct ( Yang et al., 2025 ) , and Llama 3.1 8B Instruct ( Grattafiori et al., 2024 ) . For synthesis and rubric generation, the anchor π 0 \pi_{0} is the initial policy (the same model before training). For rubric scoring, we use GPT-4o ( Hurst et al., 2024 ) as the judge π J \pi_{J} . Unless specified otherwise, we always use synthesis as the reference estimation strategy with CaT.
##### Evaluation.
On HealthBench, we report response accuracy using the held-out physician-designed rubrics, with a judge model (GPT-4o) as in the benchmark paper ( Arora et al., 2025 ) . On MATH-500, we report accuracy (fraction of problems with correct final answers). We always use G = 8 G=8 rollouts. Error bars are standard errors computed across test samples. See Appendix H for hyperparameters and Appendix I for additional experimental details.
### 4.2 Main Results
Figure 3 summarizes our main findings. CaT improves over the initial policy across all models and both domains, with improvements of up to +30% on HealthBench and +33% on MATH-500 relative to the base untrained policy.
##### CaT matches or exceeds inference-time synthesis.
On HealthBench, CaT consistently outperforms inference-time synthesis. On MATH-500, the two are comparable, with CaT slightly ahead on Gemma and Llama, and slightly behind on Qwen. Critically, CaT achieves this with 9 × 9\times less test-time compute on HealthBench (the generalization induction setting). The trained policy produces a single response, while inference-time synthesis requires G = 8 G=8 rollouts plus one generation step.
##### CaT enables improvement until rollouts converge.
Better policies generate more informative rollouts, which yield better pseudo-references, which further improve the policy. We observe this dynamic across all models as CaT surpasses its own teacher signal (the initial synthesis-generated pseudo-references) on 5 of 6 model-dataset pairs. After sufficient training, rollouts converge. Here, they agree more often, leaving less disagreement for synthesis to reconcile. At this point, the pseudo-reference provides diminishing signal over individual rollouts (Appendix D ). This is consistent with known entropy collapse in RL fine-tuning ( Yue et al., 2025 ; Song et al., 2025b ) .
### 4.3 Self-Proposed Rubrics for Non-Verifiable Domains
The central challenge in non-verifiable domains is reward derivation. We therefore compare three approaches:
• Self-proposed rubrics (ours) : Rubrics generated from the pseudo-reference.
• Physician rubrics : Human-annotated rubrics from HealthBench (oracle baseline).
• Model-as-judge : Direct LLM judgment of whether rollouts match the pseudo-reference.
##### Self-proposed rubrics compete with expert annotations.
Figure 4 (left) shows that self-proposed rubrics outperform model-as-judge on all three models and are similar to physician-annotated rubrics on two of three. The model generates evaluation criteria that are sometimes as effective as those designed by domain experts, without any human input. Poor performance with model-as-judge is likely due to the brittleness of coarse-grained judgements.
##### RL with rubrics outperforms SFT.
Figure 4 (right) compares CaT against supervised fine-tuning on synthesized pseudo-references (CaT-SFT). CaT wins on all three models, often substantially (Llama: 0.38 vs 0.28). This is consistent with prior findings that RL generalizes better than SFT from limited data ( Chu et al., 2025 ; Gunjal et al., 2026 ) .
### 4.4 Reference Estimation: Synthesis vs Selection
We compare synthesis against selection baselines for reference estimation, evaluated at inference (no RL training). This allows us to determine the best reference estimation strategies to use within the CaT framework.
##### Selection baselines.
We compare synthesis against: • Single sample : One rollout from the policy (no aggregation).
• Majority vote : Select the most common final answer across G G rollouts (verifiable domains only).
• Best-of-N (Self-BoN) : The policy selects its own best response from G G rollouts.
• Min perplexity : Select the rollout with lowest perplexity under the model, e.g., Agarwal et al. (2025) .
• Mutual predictability ( Wen et al., 2025 ) : Select the rollout with highest probability when conditioned on all other rollouts.
##### Synthesis matches or exceeds selection methods.
Figure 5 shows inference-time performance across models and datasets. Synthesis is highly effective in the non-verifiable domain, outperforming all baselines on HealthBench. It also matches or exceeds them on MATH-500, allowing it to be dropped in for use across domain types. We also provide some RL results with alternative selection methods in Appendix A.1 . We note that Llama 3.1 8B—the oldest model—shows smaller gains from synthesis over selection baselines, though sees the largest gains with RL. Synthesis requires the model to identify and resolve differences across rollouts, which weaker models may struggle with due to lower baseline capacity for meta-cognitive reasoning.
### 4.5 How Does Synthesis Differ From Selection?
Selection methods can at best recover the best rollout while generative methods, such as synthesis, can construct new responses that may exceed the best rollout. In this section, we analyze when and how this matters.
##### Synthesis disagrees with consensus.
Table 1 shows that synthesis disagrees with majority vote on 5–7% of questions and with Self-BoN on 7–10%. When synthesis disagrees, it is correct substantially more often than chance: 82–86% against majority vote, 70–82% against Self-BoN. Therefore, when it disagrees, it provides an answer that improves performance on average. These answers are useful responses that are beyond the reach of selection methods.
##### Synthesis can correct all rollouts.
On ∼ \sim 1% of questions, synthesis produces a correct answer when all rollouts are wrong. This is impossible with selection as these answers are outside of the distribution of the rollout set. Appendix B provides a worked example where rollouts contain complementary arithmetic errors. Here, synthesis integrates the correct reasoning steps while avoiding the errors.
##### Synthesis reasons over multiple rollouts.
One might ask whether synthesis simply benefits from generating one more sample instead of using the rollouts. To test this, we compare synthesis conditioned on 8 rollouts versus 1 rollout. Using Qwen 3 4B on MATH-500, with a single rollout in context, synthesis improves only marginally over that rollout (0.80 vs 0.79); with 8 rollouts, it substantially outperforms majority voting (0.85 vs 0.81). Therefore, synthesis may reconcile information across rollouts rather than act as an independent sample. Analysis of synthesis outputs in Appendix C provide further evidence.
## 5 Discussion
##### Limitations and future work.
CaT requires the base model to generate useful rollouts and, for synthesis, to reconcile them coherently; models with lower base performance see smaller gains. The framework inherits a known RL limitation: as the policy improves, rollout diversity decreases, and synthesis has less disagreement to reconcile. Training gains plateau once rollouts converge (Appendix D ). Performance may also depend on the capability of the judge model (Appendix A.3 ). Future work could generate more diverse rollouts through better sampling or exploration rewards, e.g., Song et al. (2025a) . Reference estimation and reward derivation can be improved independently. Using learned aggregators may outperform prompting-based synthesis. For reward derivation, finer-grained rubrics, e.g., partial credit, hierarchical criteria, and confidence weighting, may improve scoring accuracy and granularity. Beyond single-turn responses, CaT could extend to reasoning traces, multi-turn dialogues, or agentic trajectories.
##### Conclusion.