Transcendence: Generative Models Can Outperform The Experts That Train Them — part 3/4

Cleaned-text code points [30828, 46435). Bibliography omitted here; source.md includes it.

SQuADv2 Natural Language Temperature Denoising Experiment. We extend our analysis to the Natural Language Processing domain by running experiments on the Stanford Question Answering Dataset (SQuAD 2.0). We tested the effects of temperature denoising on the performance of several large language models (LLMs) of varying sizes. The SQuAD task involves reading comprehension and question-answering based on Wikipedia articles, making it an ideal setting to evaluate the impact of denoising on language models. We measured the exact-match, semantic-match, and F1 scores of the model outputs at different temperatures. The results show that temperature denoising leads to improved performance, corroborating the findings of our chess experiments and providing broader validation of the underlying mechanism of temperature denoising in diverse domains.

Toy Model Setting and Results. In addition, we develop a toy theoretical model to further study when transcendence is possible. This model involves a classification task with Gaussian input data and linearly separable classes. Experts label the data with noisy versions of the ground truth separator. We trained a linear model on a dataset labeled by random experts and observed the test accuracy for different temperature settings. The synthetic experiments demonstrated that transcendence occurs when expert diversity is high and temperature is low, aligning with our theoretical and empirical analysis in the chess domain.

## 5 Related Work

#### Chess and AI.

Chess has been motivating AI research since the field began. In 1950, before anyone had used the term “artificial intelligence”, automated chess were explored by both Claude Shannon [ 26 ] and Alan Turing [ 32 ] . Arguably, this history goes back even further: the famed “mechanical turk” of the 18th century was a fraudulently automated chess player. These centuries of mechanical ambitions were finally realized in 1997, when world champion Garry Kasparov was defeated by IBM’s Deep Blue [ 3 ] . Since then, chess program developers have drawn on neural approaches, with the RL-based convolutional network AlphaZero [ 27 ] far surpassing prior world champion engines such as Stockfish [ 25 ] .Our chess model testbed is inspired by a number of existing approaches, including other models trained on lichess data [ 18 ] , and other transformer-based sequential chess agents [ 23 , 5 ] .

#### Diversity beats Strength.

Another historical thread in AI research is the strength of diverse learners. Long since the development of ensemble methods that exploit learner diversity—including bagging [ 1 ] , boosting [ 6 ] , and model averaging [ 19 ] —researchers have continued to articulate this insight across settings. Similar to our chess setting, a diverse team of go playing agents have been proven and empirically shown to outperform solitary agents [ 9 ] and homogeneous teams [ 28 ] , even when the alternative models individually outperform the diverse team members [ 17 ] . We draw a connection to this deep literature through our theory, which shows that imitation learning objective and then performing low-temperature sampling subtly implies the same principle of majority voting. Teacher diversity has also been explored in the machine learning literature. One related method is ensemble distillation [ 16 ] , in which a model is trained with an additional objective to match a variety of weaker teacher models. Closer to our setting, ensemble self-training approaches [ 24 ] train a learner directly on the labels produced by varied teachers. Large language models supervised by smaller or less trained models are said to exhibit “weak to strong generalization” [ 2 ] . Overall, evidence continues to accrue that the general phenomenon we address is pervasive: that is, models can substantially improve over the experts that generate their training data.

#### Offline Reinforcement Learning.

Our work also draws connections to the Offline Reinforcement Learning [ 14 ] setting, where one attempts to learn a new policy π \pi that improves upon a fixed dataset generated by some behavior policy π β \pi_{\beta} . However, our setting of imitation learning differs substantially from this literature, as we do not explicitly train our model on a RL objective that attempts to improve upon the dataset. Importantly, such an objective oftentimes introduces training instabilities [ 15 ] and also assumes reward labels. We defer a more extended discussion of related work to Appendix D .

## 6 Discussion and Future Work

This paper introduces the concept of transcendence. Our theoretical analysis shows that low-temperature sampling is key to achieving transcendence by denoising expert biases and consolidating diverse knowledge. We validate our findings empirically by training several chess models which, under low-temperature sampling, surpass the performance of the players who produced their training data, as well as further experiments in natural language question-answering and toy Gaussian models. We additionally highlight the necessity of dataset diversity for transcendence, emphasizing the role of varied expert perspectives.

#### Limitations.

While our work provides a strong foundation for understanding and achieving transcendence in generative models, several avenues for future research remain. Future work may investigate transcendence and its causes in domains and contexts beyond chess, such as natural language processing, computer vision, and text-to-video, to understand the generalizability of our findings. Additionally, our theoretical framework assumes that game conditions at test time match those seen during training; in order to extend our findings to cases of composition or reasoning, we must forego this assumption.

#### Future Work.

Future work could also explore the practical implementations of transcendence, and ethical considerations in the broader context of deployed generative models. Ultimately, our findings lay the groundwork for leveraging generative models to not only match but exceed human expertise across diverse applications, pushing the theoretical boundaries of what generative models can achieve.

#### Broader Impact.

The possibility of “superintelligent” AGI has recently fueled many speculative hopes and fears. It is therefore possible that our work will be cited by concerned communities as evidence of a threat, but we would highlight that the denoising effect addressed in this paper does not offer any evidence for a model being able to produce novel solutions that a human expert would be incapable of devising. In particular, we do not present evidence that low temperature sampling leads to novel abstract reasoning, but just denoising of errors.

## Acknowledgements

Sham Kakade acknowledges this work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence; support from the Office of Naval Research under award N00014-22-1-2377, and the National Science Foundation Grant under award #IIS 2229881.

## Appendix A Proofs

Here we prove Proposition 1 , where transcendence cannot occur by purely using imitation learning in our setting where all experts are sampled uniformly across the input distribution.

###### Proof.

From linearity of the expectation R p test ​ ( f ^ ) \displaystyle R_{p_{\mathrm{test}}}(\hat{f}) = 𝔼 x ∼ p test ​ [ r x ​ ( f ¯ ) ] \displaystyle=\mathbb{E}_{x\sim p_{\mathrm{test}}}\left[r_{x}(\overline{f})\right] = 𝔼 x ∼ p test ​ [ 1 k ​ ∑ i = 1 k r x ​ ( f i ) ] = 1 k ​ ∑ i = 1 k R p test ​ ( f i ) ≤ max i ⁡ R p test ​ ( f i ) \displaystyle=\mathbb{E}_{x\sim p_{\mathrm{test}}}\left[\frac{1}{k}\sum_{i=1}^{k}r_{x}(f_{i})\right]=\frac{1}{k}\sum_{i=1}^{k}R_{p_{\mathrm{test}}}(f_{i})\leq\max_{i}R_{p_{\mathrm{test}}}(f_{i}) ∎

We now give the proof of Proposition 2 that if the arg-max prediction is better than the best expert, then transcendence is possible with low-temperature sampling.

###### Proof.

Observe that for all q q , it holds that lim τ → 0 softmax ⁡ ( q , τ ) = arg ⁡ max ⁡ ( q ) \lim_{\tau\to 0}\mathrm{softmax}(q;\tau)=\arg\max(q) . Therefore, for all x x lim τ → 0 r x ​ ( f ^ τ ) = lim τ → 0 ∑ y r ⁡ ( x , y ) ⋅ f ^ τ ​ ( y | x ) = ∑ y r ⁡ ( x , y ) ​ f ^ max ​ ( y | x ) = r x ​ ( f ^ max ) \lim_{\tau\to 0}r_{x}(\hat{f}_{\tau})=\lim_{\tau\to 0}\sum_{y}r(x,y)\cdot\hat{f}_{\tau}(y|x)=\sum_{y}r(x,y)\hat{f}_{\max}(y|x)=r_{x}(\hat{f}_{\max}) and so, lim τ → 0 R p test ​ ( f ^ τ ) \displaystyle\lim_{\tau\to 0}R_{p_{\mathrm{test}}}(\hat{f}_{\tau}) = lim τ → 0 𝔼 x ∼ p test ​ [ r x ​ ( f ^ τ ) ] \displaystyle=\lim_{\tau\to 0}\mathbb{E}_{x\sim p_{\mathrm{test}}}\left[r_{x}(\hat{f}_{\tau})\right] = 𝔼 x ∼ p test ​ [ lim τ → 0 r x ​ ( f ^ τ ) ] = 𝔼 x ∼ p test ​ r x ​ ( f ^ max ) = R p test ​ ( f ^ max ) \displaystyle=\mathbb{E}_{x\sim p_{\mathrm{test}}}\left[\lim_{\tau\to 0}r_{x}(\hat{f}_{\tau})\right]=\mathbb{E}_{x\sim p_{\mathrm{test}}}r_{x}(\hat{f}_{\max})=R_{p_{\mathrm{test}}}(\hat{f}_{\max}) Therefore, the required immediately follows. ∎

To prove Proposition 3 , we directly use the result in Proposition 2 .

###### Proof.

Notice that for this expert, arg ​ max ( f ( ⋅ | x ) ) = f ∗ ( y | x ) \argmax(f(\cdot|x))=f^{*}(y|x) , which achieves higher reward compared to f f . Therefore, Theorem 2 implies that we achieve transcendence in the setting where all the data is generated by a single expert f f . ∎

Next, we give the proof for that low-temperature sampling can be thought of as performing majority vote [ 1 , 6 ] between the experts:

###### Proposition 5 .

Let 𝐳 = [ z 1 , z 2 , … , z n ] \mathbf{z}=[z_{1},z_{2},\dots,z_{n}] be a vector and τ > 0 \tau>0 be a temperature parameter. Define the softmax function as σ τ ​ ( z i ) = e z i / τ ∑ j = 1 n e z j / τ . \sigma_{\tau}(z_{i})=\frac{e^{z_{i}/\tau}}{\sum_{j=1}^{n}e^{z_{j}/\tau}}. Then, as τ → 0 + \tau\to 0^{+} , the limit of the softmax function is given by lim τ → 0 + σ τ ​ ( z i ) = { 1 k , if ​ z i = z max , 0 , otherwise , \lim_{\tau\to 0^{+}}\sigma_{\tau}(z_{i})=\begin{cases}\displaystyle\frac{1}{k},&\text{if }z_{i}=z_{\max},\\ 0,&\text{otherwise},\end{cases} where z max = max 1 ≤ j ≤ n ⁡ z j z_{\max}=\max_{1\leq j\leq n}z_{j} , and k k is the number of indices i i such that z i = z max z_{i}=z_{\max} .

###### Proof.

Let z max = max 1 ≤ j ≤ n ⁡ z j z_{\max}=\max_{1\leq j\leq n}z_{j} , and define the set S = { i ∣ z i = z max } , S=\{i\mid z_{i}=z_{\max}\}, with cardinality k = | S | k=|S| .

For each i i , let Δ i = z i − z max ≤ 0 . \Delta_{i}=z_{i}-z_{\max}\leq 0. Then the softmax function becomes σ τ ​ ( z i ) = e ( z max + Δ i ) / τ ∑ j = 1 n e ( z max + Δ j ) / τ = e Δ i / τ ∑ j = 1 n e Δ j / τ , \sigma_{\tau}(z_{i})=\frac{e^{(z_{\max}+\Delta_{i})/\tau}}{\sum_{j=1}^{n}e^{(z_{\max}+\Delta_{j})/\tau}}=\frac{e^{\Delta_{i}/\tau}}{\sum_{j=1}^{n}e^{\Delta_{j}/\tau}}, since e z max / τ e^{z_{\max}/\tau} cancels out in the numerator and denominator.

We analyze the behavior of the terms as τ → 0 + \tau\to 0^{+} .

For i ∈ S i\in S we have Δ i = 0 \Delta_{i}=0 and so: e Δ i / τ = e 0 = 1 . e^{\Delta_{i}/\tau}=e^{0}=1.

For i ∉ S i\notin S we have Δ i < 0 \Delta_{i}<0 so lim τ → 0 e Δ i / τ = 0 \lim_{\tau\to 0}e^{\Delta_{i}/\tau}=0

Therefore, the denominator simplifies to lim τ → 0 + ∑ j = 1 n e Δ j / τ = ∑ j ∈ S lim τ → 0 + e Δ j / τ + ∑ j ∉ S lim τ → 0 + e Δ j / τ = ∑ j ∈ S 1 + ∑ j ∉ S 0 = k . \lim_{\tau\to 0^{+}}\sum_{j=1}^{n}e^{\Delta_{j}/\tau}=\sum_{j\in S}\lim_{\tau\to 0^{+}}e^{\Delta_{j}/\tau}+\sum_{j\notin S}\lim_{\tau\to 0^{+}}e^{\Delta_{j}/\tau}=\sum_{j\in S}1+\sum_{j\notin S}0=k.

Similarly, the numerator becomes lim τ → 0 + e Δ i / τ = { 1 , if ​ i ∈ S , 0 , if ​ i ∉ S . \lim_{\tau\to 0^{+}}e^{\Delta_{i}/\tau}=\begin{cases}1,&\text{if }i\in S,\\ 0,&\text{if }i\notin S.\end{cases}

Thus, for each i i , lim τ → 0 + σ τ ​ ( z i ) = lim τ → 0 + e Δ i / τ lim τ → 0 + ∑ j = 1 n e Δ j / τ = { 1 k , if ​ i ∈ S , 0 , if ​ i ∉ S . \lim_{\tau\to 0^{+}}\sigma_{\tau}(z_{i})=\frac{\lim_{\tau\to 0^{+}}e^{\Delta_{i}/\tau}}{\lim_{\tau\to 0^{+}}\sum_{j=1}^{n}e^{\Delta_{j}/\tau}}=\begin{cases}\displaystyle\frac{1}{k},&\text{if }i\in S,\\ 0,&\text{if }i\notin S.\end{cases}

This concludes the proof. ∎

Finally, we give the proof of Proposition 4 , or the statement that transcendence can occur from multiple experts if the test distribution p test p_{\mathrm{test}} is spread across multiple disjoing subsets of 𝒳 i \mathcal{X}_{i} .

###### Proof.

In this case, observe that for all i i R p test ​ ( f i ) \displaystyle R_{p_{\mathrm{test}}}(f_{i}) = p test ​ ( 𝒳 i ) ⋅ 𝔼 x ∼ p test | 𝒳 i ​ r x ​ ( f ∗ ) + p test ​ ( 𝒳 ∖ 𝒳 i ) ⋅ 𝔼 x ∼ p ¯ | 𝒳 ∖ 𝒳 i ​ [ 𝔼 y ∼ Uni ⁡ ( 𝒴 ) ​ r ​ ( x , y ) ] \displaystyle=p_{\mathrm{test}}(\mathcal{X}_{i})\cdot\mathbb{E}_{x\sim p_{\mathrm{test}}|_{\mathcal{X}_{i}}}r_{x}(f^{*})+p_{\mathrm{test}}(\mathcal{X}\setminus\mathcal{X}_{i})\cdot\mathbb{E}_{x\sim\overline{p}|_{\mathcal{X}\setminus\mathcal{X}_{i}}}\left[\mathbb{E}_{y\sim\mathrm{Uni}(\mathcal{Y})}r(x,y)\right] < R p test ​ ( f ∗ ) \displaystyle<R_{p_{\mathrm{test}}}(f^{*}) Therefore, we get that for all x x f ^ ​ ( y | x ) = 1 k ​ ∑ j = 1 k f j ​ ( y | x ) = k − 1 k ⋅ 1 | 𝒴 | + 1 k ​ f ∗ ​ ( y | x ) = k − 1 k ⋅ | 𝒴 | + 1 k ​ | Y x ∗ | ⋅ 𝟏 y ∈ Y x ∗ \hat{f}(y|x)=\frac{1}{k}\sum_{j=1}^{k}f_{j}(y|x)=\frac{k-1}{k}\cdot\frac{1}{\left\lvert\mathcal{Y}\right\rvert}+\frac{1}{k}f^{*}(y|x)=\frac{k-1}{k\cdot\left\lvert\mathcal{Y}\right\rvert}+\frac{1}{k\left\lvert Y_{x}^{*}\right\rvert}\cdot\mathbf{1}_{y\in Y_{x}^{*}} Thus, we get f max = f ∗ f_{\max}=f^{*} , and the required follows from Proposition 2 .

∎

## Appendix B Additional Denoising Visualizations

## Appendix C Intuition of low temperature sampling inducing transcendence

To build intuition for the primary mechanism of transcendence that we explore in this paper, we give the following toy progression of distributions in order to clearly illustrate how low-temperature sampling can induce transcendence through majority voting. Here, the middle purple action represent the correct, high-reward output, whilst the left and right actions are low-reward bad outputs. We plot the probability of each output as a label on the x axis.

## Appendix D Further Related Work

### D.1 Label Disagreement

Label disagreement in training data, in particular, can improve models in practice. Xie et al. [36] empirically show that adding random noise to teacher-generated labels can improve a student model. Uma et al. [33] even survey the literature on human interannotator disagreement and find a trend of improvements when models are trained on the full set of disagreeing labels rather than on majority vote labels or only on data where labelers agree. Our theoretical claims build on these findings by making the point that the learner can even improve on these original diverse labelers.

### D.2 Offline Reinforcement Learning

Although most Offline Reinforcement Learning algorithms train on an RL objective, perhaps most similar to our work is Decision Transformer [ 4 ] and Trajectory Transformer [ 8 ] : prior models trained on just the sequence prediction of trajectories. Most notably, Decision Transformer also finds an alternative form of transcendence than the one explored in this paper: by conditioning the trained transformer by the performance of the trajectory, at inference time they can then prompt the model to perform better than the best trajectory seen during training. This remains another promising direction to explore transcendence under.