Unsupervised Elicitation of Language Models — part 2/3

Cleaned-text code points [15760, 31622). Bibliography omitted here; source.md includes it.

Finding 4: ICM scales up with pretrained model capabilities. Since ICM focuses on elicitation, its effectiveness may naturally improve with pretrained model capabilities. We study the scaling properties of ICM on TruthfulQA and present results in Figure 5 . While ICM moderately underperforms the golden label baseline on Llama 8B, it performs comparably on LLama 70B.

We were initially very skeptical of these findings, because they seemed clearly too good to be true, and suspiciously close to training with actual labels. To ensure we didn’t accidentally train on the labels, (1) we re-ran the experiment several times on different datasets, (2) we copied the dataset into a new file, excluding any labels before re-running our algorithm with that file, and (3) one coauthor independently replicated the findings on the Claude 3.5 Haiku base model using a different codebase.

### 4.2 Unsupervised Elicitation Fails when Concepts are not Salient

To highlight some of our algorithm’s limitations, we design a task specifically to be impossible for unsupervised elicitation. Suppose we really like poems about the sun, so we construct a comparison dataset where all poems that mention the word ”sun” are preferred. The only task description we give the LMs is to judge which poem is better, but it is impossible for the LM to know our specific personal preference about poems. In other words, this task is not “salient” to pretrained models, because their understanding of the “poem quality” concept is not related to the sun. To construct the dataset, we use Claude 3.5 Sonnet to generate pairs of poems, and use designed prompts and post-filterings to ensure only one of them mentions “sun”. Experiment results with Llama 70B are shown in Figure 5 . As expected, we find ICM performs no better than random guessing.

### 4.3 Eliciting Superhuman Capabilities

After studying unsupervised elicitation on three common NLP datasets, we are further interested in tasks where pretrained models are strongly superhuman. To study this, we explore an author gender prediction task using the Blog Authorship Corpus ( Schler et al., 2006 ) . 2 2 2 Our goal is not to improve AI performance at predicting author gender, but rather to study how well this capability is already present in pretrained models.

Using pairs of blog posts ( A A and B B ) from the Blog Authorship Corpus, one written by a male and one by a female, the task is to predict which one is more likely to be written by a male. We use the simple asymmetry logical consistency: A > B A>B contradicts B > A B>A .

To build human baselines, we recruit 5 annotators to label 1) 48 training examples for prompting and 2) 100 test examples for estimating human performance on the whole test set. Human labels have perfect consistency but bad accuracy (60% on the test set, 53.8% on the training set).

As shown in Figure 6 , our method matches golden supervision (80% accuracy), significantly outperforming the estimated human accuracy (60%). In comparison, prompting with weak human labels or commercial post-training all fail to fully leverage pretrained models’ superhuman-level capability.

### 4.4 Training an Assistant Chatbot without Supervision

After verifying ICM on standard benchmarks, we investigate whether it can scale to commercial production runs and improve frontier assistant chatbots. Specifically, we aim to train a helpful, harmless, and honest chat assistant based on Claude 4 Sonnet, without introducing any external supervision labels. In these experiments, we use a scalable variant of ICM that particularly tackles long-context challenge when applied to proudction data. See Appendix G for more details.

We use the task description “Output A is more helpful, harmless, and honest than Output B” to construct 5,000 pairwise preference data. Then we use our method to generate labels with Claude 4 Sonnet pretrained models, and fine-tune it into a RM. As a baseline, we use proudction-grade human labels to train a human-supervised RM.

Using the unsupervised and human-supervised RM, we train two assistants via reinforcement learning. The training data is a mix of math, code and instruction following tasks. We train both assistant policies on 250,000 episodes. We then evaluate both policies on RewardBench, where the policy responses are scored by a production-grade Claude 4 Opus RM.

Results are shown in Figure 7 . The unsupervised assistant matches its human-supervised counterpart on average, with higher scores on chat and safety and lower scores on math and code. We suspect that this is because the production-grade human labels are of higher quality on these crisp reasoning tasks.Interestingly, RL against our unsupervised RM learns faster than RL against the human-supervised RM (e.g., 2.5x the speed on Chat and Chat-Hard).

## 5 Ablations

Comparing to randomly perturbed labels. Pretrained models may just be robust to label noise on these benchmarks, thus training labels with a certain level of noise could always match the performance of training on golden labels. To rule out this hypothesis, we construct a set of randomly perturbed labels with the same accuracy as our model-generated labels, and conduct ablation studies with Llama pretrained models with many-shot prompting. As shown in Figure 8 , our model-generated labels always achieve substantially better performance. We suspect this is because our labels are more aligned with the model’s understanding of correct labels for the task.

Evaluating robustness to worst-case initialization. It is possible that ICM could collapse under bad initialization (e.g., all initial K K labels are wrong), but we coincidentally never encounter that in Sec. 4 because it happens rarely.

We thus investigate ICM ’s robustness against different initializations, including random labels (default setting), entirely wrong labels, or golden labels. Figure 9 showcases results on TruthfulQA with the Llama 8B model. We report the test accuracy using many-shot prompting. Under random initialization, ICM achieves a comparable average accuracy but a slightly higher variance. Even under worst-case initialization, ICM remains robust, experiencing only a moderate performance drop rather than complete failure. This is mainly due to its iterative nature: a few initial bad labels would not degrade the performance significantly, as they can be gradually corrected as the algorithm progresses.

Ablating logical consistency. Logical consistency may be of limited value in ICM : we only use simple logical consistency that can be applied to many tasks, as determining fine-grained consistency relationships across examples is challenging. Empirically, we observe different impacts of logical consistency across tasks (Figure 10 ). For example, on TruthfulQA, removing logical consistency only leads to moderately worse results, as the degenerate solution of solely maximizing mutual predictability (i.e. assigning the same label everywhere) happens rarely. In contrast, logical consistency is crucial on Alpaca, since the degenerate solution almost always happens without that.

## 6 Discussion

The role of logical consistency. As Sec. 5 shows, removing consistency often does not degrade the maximal performance, but increases the variance. Specifically, the algorithm becomes more likely to collapse into degenerate solutions that have low logical consistency, like assigning the same label to all data points. Therefore, we understand mutual predictability as the most important term that leads to our empirical success. In particular, mutual predictability also likely enforces complex (probabilistic) consistencies, which cannot be easily captured by general axiomatic logical checks.

Unsupervised elicitation as an alignment method. In practice, when using unsupervised elicitation for alignment, we would still need humans in the loop for various parts of the post-training process. For example, ICM can be directly applied to enhance constitutional AI ( Bai et al., 2022b ) for aligning LMs. Specifically, for each human-specified constitution, we can replicate our pipeline in Sec. 4.4 : use ICM to label which assistant response follows the constitution more accurately and train an unsupervised reward model, then use reinforcement learning to optimize and align the assistant towards the constitution. Additionally, we still need humans to validate whether the model is interpreting the constitution as intended, for example using scalable oversight techniques ( Saunders et al., 2022 ; McAleese et al., 2024 ; Wen et al., 2024a ) .

Limitations. As shown in Sec. 4.2 , our algorithm cannot elicit any concepts or skills unless they are “salient” to the pretrained model. In addition, one potential concern is that unsupervised elicitation might be related to data contamination from pretraining. While we cannot directly verify this concern as Llama pre-training corpus is not accessible, there are several pieces of evidence that make data contamination less worrying. For example, the production assistant training data in Sec. 4.4 is certainly not involved in Claude 4 Sonnet’s pretraining corpus. See Appendix J for more discussion.

Conclusion. As LMs advance, they will become capable of doing tasks that humans struggle to evaluate. Therefore, we need new algorithms beyond RLHF to ensure that they still act in accordance with human intent. Our results suggest that unsupervised elicitation is a promising avenue to elicit specific skills from the model without being bounded by the ability of humans.

## Ethics Statement

While this paper proposes an unsupervised algorithm to elicit superhuman capabilities from LMs, this does not necessarily mean humans will lose control over LMs. As discusssed in Sec. 6 and empirically showed in Sec. 4.4 , our method could be combined with human-specified constituions to potentially align powerful LMs with human values.

## Appendix

## Appendix A The Use of LLMs

We only use LLMs to polish paper writing. We did not use LLMs to generate experimental code or directly generate the paper draft.

## Appendix B Related Work

Scaling Beyond Human Supervision. Recent work has shown diverse failure modes of post-training with unreliable human supervision. For example, LMs can learn to reward-hack human-designed supervision signals ( Baker et al., 2025 ) or even humans themselves ( Wen et al., 2024b ) . To scale beyond human supervision, one standard method is to use high-quality verifiable rewards. For example, in math, we can match model outputs with existing golden solutions ( Guo et al., 2025 ) . Unfortunately, such verifiable rewards are unavailable for most tasks. In contrast, our method can provide superhuman-level supervision in broad tasks, even including creating a general helpful, harmless, and honest assistant.

Evidence of Latent Capabilities in LMs. Recent work shows that pre-trained base models have already learned strong capabilities for downstream tasks, and post-training in fact does not add much. For example, pretrained models can achieve a comparable or even higher pass@ k k than their post-trained counterparts when k k is large enough, even when post-training is done with verifiable rewards ( Yue et al., 2025 ) . Similarly, pretrained and post-trained models perform nearly identically in decoding, while most distribution shifts occur with stylistic tokens such as discourse markers ( Lin et al., 2023 ) . When inspecting model latent representations, recent work also finds that LMs encode strong signals of reasoning correctness ( Zhang et al., 2025 ) or hallucination ( Kadavath et al., 2022 ; Ferrando et al., 2024 ) . Our work dmeonstrates that it is promising to elicit these LMs’ important latent capabilities in a fully unsupervised way.

Unsupervised Elicitation of LMs. CCS ( Burns et al., 2022 ) is one of the most representative works for unsupervised elicitation, which works by solely using simple logical consistency to find latent knowledge. While moderately outperforming the zero-shot prompting baseline, CCS still significantly underperforms supervised approaches. As argued in ( Farquhar et al., 2023 ) , CCS, as well as other unsupervised approaches, often cannot find knowledge, because there are many other prominent features that can satisfy logical consistency properties. Our method addresses this challenge by introducing mutual predictability.

Several concurrent studies explore unsupervised elicitation by minimizing label entropy ( Zhao et al., 2025 ; Agarwal et al., 2025 ) , differing from our scoring function. Empirically, these studies focus on math or coding domains using specific Qwen pretrained models. In contrast, our work demonstrates for the first time that unsupervised elicitation algorithms can match or exceed human supervision across pretrained models and a variety of crisp and fuzzy tasks — even including training a general-purpose assistant.

Unsupervised elicitation can also be thought of as a special case of weak-to-strong generalization ( Burns et al., 2023 ; Hase et al., 2024 ) : while they try to use weak human supervision to elicit strong LMs, we seek to ignore the weak human supervision altogether.

## Appendix C Illustrative Example of Scoring Function

A higher score indicates that the labels are more logically consistent and mutually predictable according to the pre-trained base model.

To provide more intuition about how our scoring function works, consider a small dataset D consisting of four examples: x 1 x_{1} : lying is more harmless than honesty

x 2 x_{2} : honesty is more harmless than lying

x 3 x_{3} : hacking unit tests is more harmless than writing correct code

x 4 x_{4} : writing correct code is more harmless than hacking unit tests

Logical consistency constraints that A > B A>B and B > A B>A cannot both be True, so y 1 ≠ y 2 y_{1}\neq y_{2} , y 3 ≠ y 4 y_{3}\neq y_{4} . According to the model’s understanding of the harmlessness concept, x 2 x_{2} and x 4 x_{4} are mutually predictable, and y 2 = y 4 = T ​ r ​ u ​ e y_{2}=y_{4}=True is more likely than y 2 = y 4 = F ​ a ​ l ​ s ​ e y_{2}=y_{4}=False . So the label assignment { F ​ a ​ l ​ s ​ e , T ​ r ​ u ​ e , F ​ a ​ l ​ s ​ e , T ​ r ​ u ​ e } \{False,True,False,True\} achieves high mutual predictability and logical consistency, yielding a high overall score.

## Appendix D Additional Implementation Details

We use Llama 3.1 8B, Llama 3.1 70B, and Claude 4 Sonnet in our experiments. Unless stated otherwise, we always use pretrained models that have received no additional training, i.e. no supervised fine-tuning on demonstrations, RLHF, RL on outcomes, or any other post-training.

### D.1 Hyperparameters

We set the initial temperature T 0 = 10 T_{0}=10 , the final temperature T min = 0.01 T_{\text{min}}=0.01 , and the cooling rate β = 0.99 \beta=0.99 . For the coefficient α \alpha , we always start with α = 50 \alpha=50 . While a large α \alpha usually yields labels of higher quality, it may excessively restrict the acceptance criteria, causing the algorithm to frequently reject new labels. Therefore, we may adjust α \alpha to a smaller value (20 or 30) based on the search speed on the training data, without reference to any validation data.

For many-shot prompting, we use as many examples as possible that can fit into the model’s context, e.g., 160 examples for Alpaca. For fine-tuning, we train the model for 3 epochs. Specifically, for Llama 8B, we do full parameter fine-tuning with a learning rate of 1e-5; for Llama 70B, we do LoRA fine-tuning with a rank of 16 and a learning rate of 5e-5.

### D.2 Data Statistics

Table 2 shows the size of train/test splits used for the experiments in Sec. 4.1 .

## Appendix E Additional Baselines

In this section, we compare ICM to several additional baselines.

### E.1 Distillation