Unsupervised Elicitation of Language Models — part 1/3

Cleaned-text code points [0, 15760). Bibliography omitted here; source.md includes it.

# Unsupervised Elicitation of Language Models

###### Abstract

To steer pretrained language models for downstream tasks, today’s post-training paradigm relies on humans to specify desired behaviors. However, for models with superhuman capabilities, it is difficult or impossible to get high-quality human supervision. To address this challenge, we introduce a new unsupervised algorithm, Internal Coherence Maximization (ICM), to fine-tune pretrained language models on their own generated labels, without external supervision . On GSM8k-verification, TruthfulQA, and Alpaca reward modeling tasks, our method matches the performance of training on golden labels and outperforms training on crowdsourced human supervision. On tasks where LMs’ capabilities are strongly superhuman, our method can elicit those capabilities significantly better than training on human labels. Finally, we show that our method can improve the training of frontier LMs: we use our method to train an unsupervised reward model and use reinforcement learning to train a Claude 4 Sonnet-based assistant. The resulting assistant matches its counterpart trained on production-grade human labels on average, with higher scores on chat and safety yet lower scores on math and coding.

## 1 Introduction

Today’s post-training paradigm of pre-trained language models (LMs) still relies on humans to specify desired behaviors, either through demonstrations or preference feedback ( Ouyang et al., 2022 ; Glaese et al., 2022 ; Bai et al., 2022a ) . However, as tasks and model behaviors grow more complex, human supervision becomes increasingly unreliable: LMs can learn to mimic mistakes in demonstrations ( Asare et al., 2023 ) or exploit flaws in feedback ( Wen et al., 2024b ) . How do we train LMs to do tasks that are too difficult for humans to demonstrate or evaluate reliably?

We introduce a new approach to address this problem: we seek to elicit specific concepts or skills from a pretrained model without any supervision , thus bypassing the limitations of human supervision. Pretrained models have already learned rich representations about important human concepts, such as mathematical correctness, truthfulness, and helpfulness ( Burns et al., 2022 ) . We should not need to teach LMs much about these concepts in post-training—instead, we can just “elicit” them from LMs.

Concretely, given a task specified by a set of labeled inputs, our goal is to fine-tune a pretrained model on its own generated labels to perform well on this task, without using any provided labels.

Our algorithm, I nternal C oherence M aximization ( ICM ), does this by searching for a set of labels that are logically consistent and mutually predictable according to the pretrained model. Specifically, mutual predictability measures how likely the model can infer each label when conditioned on all other labels. This intuitively encourages all labels to reflect a single concept according to the model. Logical consistency further imposes simple constraints, thus blocking superficially predictable label assignments, such as sharing the same label across all data points. Since finding the optimal label set that maximizes this objective is computationally infeasible, ICM uses a search algorithm inspired by simulated annealing ( Pirlot, 1996 ) to approximately maximize it.

We show that ICM matches the performance of training on golden labels on TruthfulQA ( Lin et al., 2021 ) and GSM8K ( Cobbe et al., 2021 ) , and surpasses training on crowdsourced human labels on Alpaca ( Taori et al., 2023 ) . Additionally, on a task where LMs are strongly superhuman---identifying an author’s gender from a writing sample 1 1 1 We use a widely-adopted academic dataset ( Schler et al., 2006 ) for studying AI fairness ( Coavoux et al., 2018 ; Lyu et al., 2020 ) , which consists of self-reported author information. — ICM significantly outperforms human supervision.

Beyond standard benchmarks, we investigate ICM ’s potential in improving frontier models by training a version of Claude 4 Sonnet assistant without any human supervision. Specifically, we first train a reward model (RM) with ICM , then train an assistant via reinforcement learning, which is assessed by Claude 4 Opus’s production-grade RM. Compared with the counterpart trained on production-grade human labels, our unsupervised assistant learns faster during RL and yields comparable scores on average, with higher scores on chat and safety and lower scores on math and code.

While prior work has studied unsupervised elicitation methods in simple toy settings ( Burns et al., 2022 ) , our work demonstrates for the first time that it is possible to match or exceed human supervision in realistic settings. By successfully training a Claude 4 Sonnet-based assistant without any human labels and achieving comparable performance to its human-supervised counterpart, we show that unsupervised elicitation is practically useful for post-training frontier models into general assistants.

## 2 Methodology

### 2.1 Problem Statement

Typically, fine-tuning LMs for a task requires a labeled dataset D = { ( x i , y i ∗ ) } D=\{(x_{i},y_{i}^{*})\} . However, for many complex tasks, obtaining externally human-specified { y i ∗ } \{y_{i}^{*}\} is difficult or impossible. Therefore, our goal is to use the LM to estimate labels { y i } \{y_{i}\} , based purely on the inputs { x i } \{x_{i}\} .

In particular, we are mainly focused on classification tasks (e.g. reward modeling) in this paper, as we can naturally use reinforcement learning to optimize for open-ended generation tasks. We demonstrate this by training a version of genreal Claude 4 Sonnet assistant in Sec. 4.4 .

In this following section, we explain how an LM can internally score the quality of { y i } \{y_{i}\} , without referencing external labels { y i ∗ } \{y_{i}^{*}\} , and how to algorithmically maximize this score.

### 2.2 Scoring Function

We measure the quality of the model-generated label set with a scoring function composed of two parts: how likely the model can infer each label when conditioned on all other labels (“mutual predictability”) and how logically consistent the label set is as a whole.

Mutual Predictability. As illustrated in the top panel of Figure 2 , given a pretrained model P θ P_{\theta} , for each example x i x_{i} , we calculate the probability of its label y i y_{i} by conditioning all other | D | − 1 |D|-1 labels (e.g. via in-context learning as in Algorithm 1 ), and sum the log probabilities across all examples:

𝒫 θ ​ ( D ) = ∑ i = 0 N log ⁡ P θ ​ ( y i | x i , D ∖ ( x i , y i ) ) \mathcal{P}_{\theta}(D)=\sum_{i=0}^{N}\log P_{\theta}(y_{i}|x_{i},D\setminus(x_{i},y_{i}))

Intuitively, this yields a high score if { ( x i , y i ) } \{(x_{i},y_{i})\} collectively specify a single coherent concept according to P θ P_{\theta} , i.e. a labeling scheme where P θ P_{\theta} can confidently infer any label y i y_{i} from the others. Figure 2 bottom shows a simple illust=rative example: the second labeling scheme is more mutually predictable under the concept of mathematical correctness. See Appendix C for another example.

However, mutual predictability alone allows some degenerate solutions, e.g. assigning the same label to all data points can artificially inflate P θ ​ ( D ) P_{\theta}(D) as well.

Logical Consistency. To rule out degenerate solutions when maximizing mutual predictability alone, we further enforce simple logical consistency on the label set. Specifically, we are given a logical consistency function c ⁡ ( x i , y i , x j , y j ) ∈ { 0 , 1 } c(x_{i},y_{i},x_{j},y_{j})\in\{0,1\} , which is an indicator function that checks whether the labels y i y_{i} and y j y_{j} on data points x i x_{i} and x j x_{j} are logically consistent with each other. We use it to measure inconsistencies in our labels: ℐ ⁡ ( D ) = ∑ i = 1 | D | ∑ j = 1 | D | c ⁡ ( x i , y i , x j , y j ) \mathcal{I}(D)=\sum_{i=1}^{|D|}\sum_{j=1}^{|D|}c(x_{i},y_{i},x_{j},y_{j})

Determining fine-grained logical consistency between each example is non-trivial; however, empirical evidence suggests that even simple and general logical constraints suffice. For example, when judging mathematical correctness, two different answers cannot both be True. Another general logical constraint is asymmetry: when comparing two LM outputs, A > B A>B and B > A B>A cannot both be True.

Overall Scoring Function. Combining the two terms, our scoring function is defined as follows: U ⁡ ( D ) = α ⋅ 𝒫 θ ​ ( D ) − ℐ ⁡ ( D ) U(D)=\alpha\cdot\mathcal{P}_{\theta}(D)-\mathcal{I}(D) where α \alpha is a hyperparameter to balance the strength of mutual predictability and logical consistency.

### 2.3 Our Algorithm

Finding the optimal label set that maximizes our scoring function is an integer programming problem, which is computationally infeasible for realistic dataset sizes ( | D | > 10 3 |D|>10^{3} ). ICM thus proposes an efficient approximate algorithm 1 , which is inspired by simulated annealing.

Starting from an empty labeled set, ICM initializes the search process with K K randomly labeled examples, then iteratively adds labels, one at a time. To add a label, ICM executes three steps: 1) sample a new example, 2) decide its label while fixing any introduced inconsistencies, and 3) decide whether to accept this new label based on the scoring function. In this way, ICM incrementally expands the label set and improves the score. The bottom of Figure 2 illustrates this iterative process.

Initialization. We initialize the searching process with K K randomly labeled examples. The choice of K K presents a trade-off. A large K K introduces significant initial noise that hinders subsequent convergence. Preliminary results show that initializing all K = | D | K=|D| examples with random labels or zero-shot predictions often traps the model in a poor initialization. Conversely, K = 0 K=0 reduces to a zero-shot setting, where the model lacks sufficient context to understand the task and achieves near-random performance. Empirically, we find that a small number (e.g., K = 8 K=8 ) often strikes a good balance by providing sufficient demonstrations while reducing initial noise ( Min et al., 2022 ) .

Choose a New Example to Label. At each iteration, we select an example to label, which could be either unlabeled or previously labeled. This allows us to dynamically correct earlier mistakes. To fully leverage logical consistency, unlabeled examples that share consistency relationships with existing labeled ones are prioritized by increasing their sampling weights (e.g., by a factor of 100).

Fix Inconsistencies. Although U ⁡ ( D ) U(D) explicitly penalizes logical inconsistencies, simply maximizing U ⁡ ( D ) U(D) during search still results in substantial label inconsistencies. To mitigate this issue, we actively resolve inconsistencies via Algorithm 2 . Specifically, when an inconsistency between a labeled data pair ( x i , x j ) (x_{i},x_{j}) arises, the algorithm checks all consistent label options for them and selects the combination that maximizes U ⁡ ( D ) U(D) . Importantly, after introducing a new label, we first fix its introduced inconsistencies and then measure U ⁡ ( D ) U(D) . Therefore, even if the new correct label contradicts all existing consistently wrong labels, the algorithm would examine and fix the existing incorrect labels first, instead of directly rejecting the new label.

Accept a New Label. We directly accept the new label if it improves U ⁡ ( D ) U(D) . Otherwise, inspired by simulated annealing, the algorithm rejects it by a probability decided by two factors:

• Score Difference Δ \Delta : Larger decreases in U ⁡ ( D ) U(D) increase the likelihood of rejection.

• Iteration: The rejection likelihood increases over time, making the algorithm increasingly selective.

## 3 Experiment Setup

### 3.1 Datasets

• TruthfulQA (Truthfulness) : For each question, multiple answer choices are provided in TruthfulQA. The task is to classify each answer choice as correct or incorrect.

• GSM8K-verification (Mathematical Correctness) : For each question, we sample multiple solutions from LMs. The task is to classify each solution as correct or incorrect. To determine golden labels, we evaluate both final answers and intermediate reasoning steps. Specifically, we prompt Claude 3.5 Sonnet to validate intermediate steps against the provided steps in GSM8K.

• Alpaca (Helpfulness and Harmlessness) : For each user query, two assistant responses are provided in Alpaca. The task is to classify which response is more helpful and harmless.

See Table 1 for dataset examples. Regarding logical consistency checks, for GSM8K and TruthfulQA, we use “two different answers cannot both be true”. For Alpaca, we use ” A > B A>B contradicts B > A B>A ”. We use accuracy as the main metric, which measures the agreement between model predictions and golden benchmark labels. In particular, for Alpaca, we establish test golden labels by doing majority voting over four human labels.

### 3.2 Baselines

We adopt the following four baselines in our main experiments. Appendix E compares ICM with more baselines (e.g. distilling from GPT-4o), and ICM consistently yields better performance.

• Zero-shot indicates zero-shot prompting on pretrained models. In particular, we use a highly optimized prompt that has been used for Anthropic’s pretrained models ( Askell et al., 2021 ) . which converts pretrained models into general assistants, significantly improving zero-shot performance.

• Zero-shot (Chat) indicates zero-shot prompting on commercial chat models, which have been through heavily optimized post-training. For example, Llama 2 chat models are post-trained on nearly 30K human demonstrations and 3 million human preference feedback ( Touvron et al., 2023 ) .

• Golden Label indicates many-shot prompting or fine-tuning with golden labels. We use as many examples as possible that can fit into the model’s context, e.g., 160 examples for Alpaca.

• Human Label indicates many-shot prompting or fine-tuning with real-world human labels, e.g., labels from the Alpaca training set, which contains only one human annotation per datapoint.

## 4 Experiments

### 4.1 Eliciting Capabilities on Common NLP Tasks

Finding 1: ICM matches the ceiling performance of golden supervision. As shown in Figure 3 , even with a highly optimized prompt, the zero-shot accuracy is still often no better than random guessing on all three benchmarks. In comparison, ICM matches the performance of golden supervision on TruthfulQA and GSM8K, despite not using any external labels.

Finding 2: ICM beats crowdsourced human supervision. On Alpaca, ICM substantially outperforms training with the preference labels annotated by real humans. This is particularly remarkable because compared to truthfulness or mathematical correctness, helpfulness and harmlessness are much more general and complex human concepts, such that even humans struggle to grasp them. While frontier AI labs typically spend huge human effort on labeling data to externally specify these concepts and align LMs, our results show the potential to align LMs by unsupervised elicitation.

Finding 3: ICM beats post-trained chat models. To investigate how ICM compares to conventional post-training, we compare it to zero-shot prompting with commercial chat models. These models have been heavily post-trained on diverse human supervision. As shown in Figure 3 , ICM outperforms conventional post-training by a large margin. Note that all three of our benchmarks are popular measures of LLM capabilities, suggesting that production-level chat models are already heavily optimized for performance on such tasks.