Unsupervised Elicitation of Language Models — part 3/3
Cleaned-text code points [31622, 40710). Bibliography omitted here; source.md includes it.
We use zero-shot prompting with commericial LMs to generate labels and train models. Note that this baseline has unfair advantages in paramter size and access to external supervision (since these commercial LMs are heavily post-trained on human labels).
Specifically, following prior work ( Huang et al., 2022 ; Prasad et al., 2024 ; Jiao et al., 2024 ) , for each example, we use GPT-4o to sample K = 10 K=10 labels and do majority-voting to decide the final label. We then fine-tune Llama models on these labels. We show the results in Table 3 .
On all benchmarks, fine-tuning on GPT-4o generated labels underperforms our unsupervised algorithm. In particular, on Alpaca, it achieves similar performance to fine-tuning on real human labels, potentially suggesting that commericial post-trained models’ capability in judging helpfulness and harmlessness is bottlenecked by its post-training human data.
### E.2 CCS
For each benchmark, we train a linear probe on model activations using CCS ( Burns et al., 2022 ) with the same hyperparameters as in the original paper. Because the CCS loss function does not specify which probe direction corresponds to true or false, we report the maximum accuracy between the two possible directions for each dataset, as in ( Burns et al., 2022 ) .
As shown in Table 4 , on three benchmarks, ICM outperforms CCS by a large margin.
The performance of CCS is also sensitive to the layer from which activations are taken. We show the benchmark performance for different layers in Figure 11 .
## Appendix F Compute Costs
ICM is one form of inference-time scaling. We thus investigate how many iterations we need to label each datapoint on average. Specifically, we report the statistics based on labeling n = 128 n=128 datapoints. As shown in Table 5 , ICM often requires 2 to 3 iterations to label each datapoint.
## Appendix G Scalable ICM
Our algorithm 1 has two scalability limitations. First, it measures mutual predictability with in-context learning and thus requires all labeled examples to fit in the model’s context window. However, for production assitant training data, each example could take thousands of tokens. Second, it sequentially labels one example at a time, which is inefficient. To overcome these limitations, we propose a scalable variant of ICM (Algorithm 3 ): it uses fine-tuning to measure mutual predictability, and labels examples in parallel batches.
Measure mutual predictability. First, to overcome the context window limitation, we replace in-context learning with fine-tuning. However, since mutual predictability is based on the probability of each label conditioned on all other | D | − 1 |D|-1 labels, measuring it directly would require fine-tuning | D | |D| individual models, which is expensive as we scale up D D . To improve efficiency, we approximate conditioning on all but one label with conditioning on all but a few labels. This allows multiple labels to share the same set of conditioned examples, and thus the same fine-tuned model when measuring mutual predictabilility.
Specifically, we randomly partition D D into F F disjoint subsets, i.e., D = ∪ D 1 , ⋯ D F D=\cup D_{1},\cdots D_{F} . Let t i t_{i} denote the subset that ( x i , y i ) (x_{i},y_{i}) belongs to. We approximate P θ ( y i | x i , D ∖ ( x i , y i ) ) P_{\theta}(y_{i}|x_{i},D\setminus(x_{i},y_{i})) with P θ ( y i | x i , D ∖ D t i ) P_{\theta}(y_{i}|x_{i},D\setminus D_{t_{i}}) .
To search for mutually predictable labels, for each fold D f D_{f} , we train one model on D ∖ D f D\setminus D_{f} and use it to relabel examples in D f D_{f} . In this way, searching for mutually predictable labels only needs | D | / F |D|/F finetuning runs (parallel) and | D | |D| parallel zero-shot inference.
Enforce logical consistency. Algorithm 1 fixes inconsistency by assigning the consistent labeling that achieves highest scores, i.e., maximizing mutual predictability and consistency. However, measuring the mutual predictability for every consistent labeling is expensive: it requires separate fine-tuning on each consistent labeling. We introduce a simpler algorithm to fix inconsistency. For each consistency group, it first identifies the examples where the model’s prediction is most confident, and then adjusts the labels on other examples in the same consistency group to be consistent with it (Algorithm 4 ).
## Appendix H Evaluation Results of Claude 4 Sonnet Reward Models
We experiment with Claude 3.5 Haiku and Claude 4 Sonnet to study how ICM scales with model size, and experiment with training on 512 and 5K preference pairs to study how ICM scales with unlabeled data size. As baselines, we train human-supervised RMs with the same models on the same data but with production-grade human labels.
We evaluate reward models (RMs) on Rewardbench ( Lambert et al., 2024 ) . Results are shown in Figure 12 . ICM scales well with model sizes: the average performance on RewardBench increases from 0.63 to 0.74 when training on 512 examples, from 0.69 to 0.79 when training on 5K examples.
Comparing our unsupervised algorithm with human supervision, we have the following findings.
In a low-data regime where human labels are too expensive to collect 3 3 3 We are particularly interested in eliciting capabilites on these challenging tasks in this paper, as using AIs to assist humans on these tasks would be highly valuable. , ICM outperforms human supervision by a large margin. For example, our unsupervised RM trained on 5K unlabeled data outperforms training on 512 human labels by 15.2% with Claude 3.5 Haiku, and by 6% with Claude 4 Sonnet on average. Note that unsupervised algorithm can be trained on unlimited amount ( > > 5 K >>5K ) of unlabeled data, so the performance gain is likely to further improve.
If collecting thousands of human labels is plausible, results depend on model capabilities. For weak models like Claude 3.5 Haiku, ICM can slightly outperform training with human supervision. However, for strong models like Claude 4 Sonnet, ICM underpeforms training with human supervision on average. Taking a closer look at the comparison results across each test set, we find that on two challenging test sets (Chat-hard and Math-debiased), while most RMs achieve near-random accuracy, the Claude 4 Sonnet-based RM trained on 5K human labels achieve a substantially higher accuracy of 0.66. Overall, since the performance of unsupervised algorithms would be bottlenecked by LMs’ existing latent capababilities, it is unsurprising that unsupervised algorithms would underpeform training on high-quality external labels in certain cases (e.g. on crisp tasks like mathematical reasoning). However. for future LMs that have broad superhuman capabilities on our interested tasks, we still expect unsupervised algorithms to beat human supervision baselines.
## Appendix I Human Annotation
In Sec. 4.3 , we study an author gender prediction task. To establish a human baseline, we recruit 5 annotators from upwork.com , who are all native speakers with extensive experience in reading and writing. Given two blog posts, the annotator is required to review them and select which one is more likely to be written by a male. Overall, we collect 5 human labels for each example.
## Appendix J Discussion: Data Contamination
While we cannot directly check data contamination since we don’t have access to Llama pre-training corpus, there are several pieces of evidence that make data contamination less worrying. 1. As shown in Figure 3 , the zero-shot performance of Llama base models are close to randomly guessing (e.g. 60% on TruthfulQA, 50% on ALpaca, and 48% on GSM8K)
2. We reformat GSM8K and TruthfulQA into classification tasks, which is differnt from the original data.
3. Most of our experiments are based on llama models. ( ) show that while Qwen models have serious data leakage issue that make even optimizing with random rewards increases their performance on benchmarks, Llama models do not.
4. In Sec. 4.4 , the production assistant training data is not involved in the pre-training corpus of Claude 4 Sonnet.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .