Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision — part 1/3

Cleaned-text code points [0, 15633). Bibliography omitted here; source.md includes it.

# Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision

###### Abstract

Where do learning signals come from when there is no ground truth in post-training? We show that inference compute itself can serve as supervision. By generating parallel rollouts and converting them into reference estimates, models can learn without human labels—critically, even in non-verifiable domains like healthcare guidance where no programmatic checker exists. We call this framework Compute as Teacher (CaT) and it turns inference-time compute from parallel rollouts into supervision for RL training. The framework has two components: (1) reference estimation which aggregates rollouts into a pseudo-reference answer, and (2) reward derivation which converts that pseudo-reference into RL rewards. For (1), we explore a simple method we call synthesis , but the framework admits any aggregator. For (2), we introduce self-proposed rubrics for non-verifiable domains. These are binary, auditable criteria generated from the pseudo-reference and scored by an LLM judge. On HealthBench, models trained with CaT match or exceed inference-time aggregation quality while using 9× less test-time compute. Here, CaT also competes with learning from expert physician annotations, yielding up to +30% relative improvement over the initial policy. The framework extends naturally to verifiable rewards, matching the best existing baselines on MATH-500 in test-time RL and demonstrating ‘drop-in’ versatility across both types of domains.

## 1 Introduction

Post-training large language models for specialized skills typically relies on supervised fine-tuning with labeled reference answers ( Ouyang et al., 2022 ; Wei et al., 2022 ) , or reinforcement learning with verifiable rewards from programmatic checkers. Such programmatic checkers are only applicable in narrow domains like math or code where formal correctness is computable ( Lambert et al., 2025 ; Shao et al., 2024 ) . Meanwhile, many valuable tasks also have no annotated reference answers. In non-verifiable settings, i.e., where answers are qualitative , such as clinical or lifestyle guidance ( Arora et al., 2025 ) , freeform dialogue ( Roller et al., 2020 ) , and creative writing ( Paech, 2023 ) , there may be multiple valid answers; experts can disagree, and deterministic rule-checking is impractical. As a result, practitioners often fall back on (i) annotation pipelines that are hard to scale, or (ii) judge-only feedback where another LLM assigns coarse scores to freeform outputs, despite known issues with inconsistency, verbosity bias, and reward hacking.

Instead, this paper asks a simple question: Can inference compute substitute for missing supervision?

##### Compute as Teacher (CaT).

We answer yes . We propose CaT, a framework that converts inference compute into supervision via two components: reference estimation and reward derivation . Crucially, parallel rollouts sample the model’s uncertainty as they differ precisely where the model is unsure. One rollout might have a correct intermediate step, another the right final answer, a third a useful verification. Collectively, the set contains more information than any individual rollout. Reference estimation aggregates this distributed information into a single pseudo-reference; reward derivation scores rollouts against the pseudo-reference for RL. Notably, the framework is agnostic to the aggregation method. Selection strategies (majority vote, best-of-N) recover the best rollout while generative strategies construct new responses by combining fragments across rollouts. We explore synthesis , a simple strategy where a frozen LLM policy conditions on the rollout set and constructs a new response, but any aggregator plugs into the same reward derivation mechanism. Since group RL methods like GRPO already generate parallel rollouts for advantage estimation, the reference estimation step adds little overhead.

##### Reference-free rewards for non-verifiable domains.

The second component, reward derivation, differs by domain. In verifiable settings (e.g., math and code), we simply check agreement with the pseudo-reference, i.e., does the rollout’s answer match? The harder problem is non-verifiable domains, where no such check exists. Here we introduce another key contribution— self-proposed rubrics —where the model generates binary criteria that characterize the pseudo-reference (e.g., “recommends consulting a medical professional,” “raises the patient’s arrhythmia concern”). An independent LLM judge checks rollouts against each criterion, and the reward is the fraction satisfied. This allows us to reward each rollout without a ground truth and enables reference-free training with RL in non-verifiable domains.

##### Why it works (intuition).

Rollouts sample from the model’s uncertainty as when they disagree, they expose where the model hedges. Synthesis forces the model to examine that disagreement and either commit to one view with evidence or construct a reconciliation. In contrast, selection can only pick one existing answer. For non-verifiable domains, rubrics ask “does this response satisfy criterion X?” rather than “is this response good?” making verification easier as decomposition turns a fuzzy judgment into a sum of fine-grained binary checks. Critically, self-proposing rubrics based on the pseudo-reference means that CaT can provide rewards for rollouts in non-verifiable domains without relying on any human annotated reference answers.

##### Practicality.

The framework is practical and drop-in. It requires no human labels, no domain-specific verifiers, and reuses group sampling compute already spent in standard RL. CaT amortizes inference-time aggregation into model weights, matching or exceeding aggregation performance while reducing per-query test-time compute by up to 9×. The trained policy’s responses also often surpass the initial aggregated pseudo-references in quality and demonstrate continued improvement until rollouts converge to similar responses. We validate across three model families (Gemma 3 4B, Qwen 3 4B, Llama 3.1 8B) on HealthBench (non-verifiable) and MATH-500 (verifiable).

##### CaT bridges several lines of work.

Like self-training ( Schmidhuber, 2003 ; Schmidhuber, 2013 ; Silver et al., 2016 ; Silver et al., 2018 ) and knowledge distillation ( Hinton et al., 2015 ) , it learns from model-generated supervision, but it derives the target by reconciling multiple samples rather than trusting a single self-label. Like test-time scaling methods (majority vote, best-of-N), it leverages parallel rollouts, but amortizes the improvements into weights via RL. Compared to LLM-as-a-judge rewards ( Zheng et al., 2023 ) , rubric-based scoring yields decomposed, specific criteria that mitigate instability and bias ( Dineen et al., 2025 ; Gunjal et al., 2026 ) . Finally, CaT complements programmatic verification ( Lambert et al., 2025 ) by extending learning to non-verifiable domains where formal checkers are unavailable.

Contributions: 1. Self-proposed rubrics for non-verifiable RL. A reward mechanism that decomposes judgment into binary criteria, enabling reference-free RL in domains where no programmatic checker exists.

2. Compute as Teacher (CaT). A general paradigm for converting inference compute into supervision via reference estimation and reward derivation, with synthesis as one effective reference estimator.

3. Comprehensive empirical study. Gains on HealthBench (non-verifiable) and MATH-500 (verifiable) across three models, with trained models matching or exceeding inference-time quality at 9× less compute.

##### Organization.

Section 2 contextualizes CaT among related work. Section 3 formalizes CaT and the self-proposed rubric mechanism. Section 4 details experimental setup. Section 4.2 presents results, ablations, and further analyses. Section 5 discusses limitations, future work, and concludes.

## 2 Related Work

##### Reference-Free Fine-Tuning.

Self-generated supervision has a long history in LLM training. Constitutional AI ( Bai et al., 2022 ) trains on self-revised generations, Self-Instruct ( Wang et al., 2023b ) generates and filters its own instruction-following data, and Quiet-STaR ( Zelikman et al., 2024 ) learns to produce reasoning tokens without reference traces. These methods target specific capabilities (harmlessness, instruction-following, reasoning). CaT is more general as it enables RL without human references in domains where models can generate diverse rollouts and, if not verifiable, where rubric-based judgment is reliable.

##### Reference-Free RL.

Recent work has explored RL without ground-truth labels. TTRL ( Zuo et al., 2025 ) uses majority-vote consensus as pseudo-labels for math and Absolute Zero ( Zhao et al., 2025a ) employs self-play on self-posed problems in math and code. These methods are effective but restricted to verifiable domains where correctness can be checked programmatically. A parallel line of work minimizes entropy or maximizes self-certainty ( Zhao et al., 2026 ; Agarwal et al., 2025 ; Prabhudesai et al., 2025 ; Gao et al., 2025 ; Li et al., 2025 ) . In particular, Wen et al. (2025) score multiple-choice answers via mutual predictability. These approaches are discriminative as they select or score existing outputs. CaT may use any of these methods as reference estimators, but it can also go beyond selection. Synthesis, if chosen as the reference estimation strategy, can construct pseudo-references outside of the rollout distribution. More critically, CaT extends to non-verifiable domains via self-proposed rubrics. Prior methods have not been applied in such settings. When we adapt them as baselines, they underperform CaT (Section 3.1 ).

##### Test-Time Scaling.

Parallel rollouts have long been used to improve inference-time performance. Majority voting or self-consistency ( Wang et al., 2023a ) selects the most common answer; best-of-N selects by confidence, perplexity, or LLM judgment ( Zheng et al., 2023 ) . These methods improve accuracy but do not transfer improvements into weights as each deployment pays the full G G -rollout cost, where G G is the number of rollouts sampled. CaT uses the same parallel rollouts to generate supervision for RL, amortizing inference compute into training. After training, the model matches or exceeds aggregation quality at 1 / G 1/G the test-time cost. Therefore, most of the compute is spent once during training, rather than repeatedly at inference time.

##### Non-Verifiable RL.

When programmatic verification is impossible, how do you derive rewards? Existing approaches assume access to reference answers. VeriFree ( Zhou et al., 2026 ) , JEPO ( Tang et al., 2025 ) , and RLPR ( Yu et al., 2025 ) compute the probability of a reference under the model’s reasoning chain. Rubrics as Rewards ( Gunjal et al., 2026 , RaR;) constructs evaluation rubrics from reference answers, then scores outputs via LLM judgment, demonstrating that rubric-based rewards outperform direct LLM-as-judge scoring. We build on this insight but remove the reference dependency entirely as CaT self-proposes rubrics from pseudo-references estimated from rollouts. Here, inference compute generates the pseudo-reference, the pseudo-reference generates rubrics, and rubrics generate rewards. Therefore, no human annotation enters the pipeline.

## 3 Compute as Teacher (CaT)

CaT converts inference compute into supervision via two components: reference estimation and reward derivation . Figure 1 illustrates the framework.

Given a prompt q q , the current policy π t \pi_{t} generates G G parallel rollouts o 1 : G o_{1:G} . These rollouts sample the model’s uncertainty as they differ precisely where the model is unsure. Reference estimation aggregates them into a pseudo-reference s s , which is a single response that serves as the supervision target. Reward derivation then scores each rollout o i o_{i} against s s , producing rewards R ⁡ ( o i , s ) R(o_{i};s) for RL.

The framework is agnostic to the aggregation method. Selection strategies (majority vote, best-of-N) choose one rollout as s s . Generative strategies like synthesis construct a new s s conditioned on all rollouts, potentially combining correct fragments scattered across the set. We explore synthesis empirically, but the reward derivation machinery—particularly self-proposed rubrics for non-verifiable domains—works with any reference estimation strategy.

##### Notation.

We write q q for the prompt, o i o_{i} for a rollout, s s for the pseudo-reference, ℛ = { r j } \mathcal{R}=\{r_{j}\} for a rubric (a set of binary criteria), π t \pi_{t} for the current policy at training step t t , π 0 \pi_{0} for the frozen anchor (the initial policy), and π J \pi_{J} for the judge model used in rubric scoring.

### 3.1 Reference Estimation

Reference estimation aggregates G G rollouts into a single pseudo-reference s s . Any aggregation strategy is compatible with the framework. Examples include:

• Majority vote : s s is the most common final answer among o 1 : G o_{1:G} ( Zuo et al., 2025 ) . Applicable only in verifiable domains where answers can be compared.

• Best-of-N : s = arg ⁡ max o i ⁡ f ⁡ ( o i ) s=\arg\max_{o_{i}}f(o_{i}) for some scoring function f f . This could be perplexity under the model ( Agarwal et al., 2025 ) , confidence heuristics ( Li et al., 2025 ) , or LLM judge scores.

• Synthesis : A frozen anchor π 0 \pi_{0} conditions on the rollout set and generates a new response that reconciles information across rollouts according to some prompt.

Selection methods can recover at most the best rollout. Synthesis can construct responses that combine correct fragments from multiple rollouts, potentially exceeding all of them. We focus on synthesis, though we compare against selection baselines in Section 4.4 .

##### Synthesis.

Given rollouts o 1 : G ∼ π t ( ⋅ ∣ q ) o_{1:G}\sim\pi_{t}(\cdot\mid q) , the anchor generates a pseudo-reference s ∼ π 0 ( ⋅ ∣ p syn , o 1 : G ) s\sim\pi_{0}\!\left(\cdot\mid p_{\text{syn}},o_{1:G}\right) (1) where p syn p_{\text{syn}} is a prompt instructing the model to reconcile the rollouts into a single, improved response. We provide p syn p_{\text{syn}} in Appendix F .

Two design choices merit explanation. First, we omit the original prompt q q from the anchor’s input for simplicity (ablation in Appendix A.4 ). Second, we use a frozen anchor (the initial policy π 0 \pi_{0} ) rather than the current policy π t \pi_{t} . This decouples exploration from estimation as π t \pi_{t} explores and improves through RL while π 0 \pi_{0} provides reference estimates. We optimize only π t \pi_{t} .

##### When does synthesis help?

Rollouts sample the model’s uncertainty since they disagree where the model hedges. A rollout might have a correct intermediate step but wrong conclusion while another might have the right answer via flawed reasoning. Synthesis examines the disagreement and can integrate complementary evidence such as correct steps from one rollout, and valid checks from another. Empirically, synthesis produces answers that disagree with the majority on 5–15% of questions and can disagree with all rollouts while still being correct (Section 4.5 ), indicating it performs a form of structured generative reconciliation rather than only rewriting answers and acting as a selector.

### 3.2 Reward Derivation

Given a pseudo-reference s s , we derive per-rollout rewards R ⁡ ( o i , s ) R(o_{i};s) for RL training. The mechanism differs by domain.

#### 3.2.1 Verifiable Domains