Transcendence: Generative Models Can Outperform The Experts That Train Them — part 1/4

Cleaned-text code points [0, 15039). Bibliography omitted here; source.md includes it.

# Transcendence: Generative Models Can Outperform The Experts That Train Them

###### Abstract

Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on. Therefore, when trained on data generated by humans, we may not expect the artificial model to outperform the humans on their original objectives. In this work, we study the phenomenon of transcendence : when a generative model achieves capabilities that surpass the abilities of the experts generating its data. We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset. 1 1 1 To play with our models, code, and data, please see our website at https://transcendence.eddie.win . We theoretically prove that transcendence can be enabled by low-temperature sampling, and rigorously assess this claim experimentally. Finally, we discuss other sources of transcendence, laying the groundwork for future investigation of this phenomenon in a broader setting.

## 1 Introduction

Generative models (GMs) are typically trained to mimic human behavior. These humans may be skilled in their various human objectives: answering a question, creating art, singing a song. The model has only one objective: minimizing the cross-entropy loss with respect to the output distribution, thereby adjusting it to match the distribution of human labels 2 2 2 Although chatbots are subject to a variety of post-training tuning methods, e.g., RLHF, we restrict our scope by assuming that the specialized knowledge and capacities are already provided by cross-entropy loss. . Therefore, one might assume the model can, at best, match the performance of an expert on their human objectives. Is it possible for these models to surpass—to transcend —their expert sources in some domains?

We illustrate an example of such transcendence in Figure 1 , which measures the chess ratings (Glicko-2 [ 7 ] ) of several transformer [ 35 ] models. Our experimental testbed is generative modeling on chess, which we choose as a domain for its well-understood, constrained nature. The transformer models are trained on public datasets of human chess transcripts, autoregressively predicting the next move in the game. To test for transcendence , we limit the maximal rating of the human players in the dataset below a specified score. We find that ChessFormer 1000 1000 and ChessFormer 1300 1300 (the latter number being the maximum rating seen during training) achieve significant levels of transcendence, surpassing the maximal rating seen in the dataset. Our focus is this capacity of a GM to transcend its expert sources by broadly outperforming any one expert. The key to our findings is the observation that GMs implicitly perform majority voting over the human experts. As these models are trained on a collection of many experts with diverse capacities, predilections, and biases, this majority vote oftentimes outperforms any individual expert, a phenomena that is known as “wisdom of the crowd”.

Our objective is to formalize the notion of transcendence and focus narrowly on this source of improvement over the experts: the removal of diverse human biases and errors. We prove that this form of denoising is enabled by low-temperature sampling, which implicitly induces a majority vote. Our result draws a subtle but deep connection from our new setting to a rich prior literature on model ensembling [ 1 , 6 , 19 ] , enabling several key results. We precisely characterize the conditions under which transcendence is possible, and give a rigorous theoretical framework for enabling future study into the phenomenon. To test the predictive power of our theory, we then empirically demonstrate these effects. Digging deeper into the effects of majority voting, we show that its advantage is primarily due to performing much better on a small subset of states—that is, under conditions that are likely key to determining the outcome of the game. We also find that diversity in the data is a necessary condition for practically effective majority voting, confirming our theoretical findings. In short:

• We formalize the notion of transcendence in generative models (Section 2 ).

• We find a key insight explaining one cause of transcendence by connecting the case of denoising experts to model ensembling. In low temperature sampling settings, we prove that a generative model can transcend if trained on a single expert that makes mistakes uniformly at random. We then extend this result to transcending a collection of experts that are each skilled in different domains (Section 3 ).

• We train a chess transformer on game transcripts that only include players up to a particular skill level. We confirm our theoretical prediction that this model only surpasses the maximum rating of its expert data generators at low temperature settings (Section 4 ).

• We visualize the distribution of changes in reward by setting a lower sampling temperature, attributing the increased performance to large improvements on a relatively small portion of states (Section 4.2 ).

• We explore the necessity of dataset diversity, and the inability of ChessFormer to transcend when trained on less diverse datasets (Section 4.2 ).

## 2 Definition of Transcendence

Denote by 𝒳 \mathcal{X} the (variable-length) input space and by 𝒴 \mathcal{Y} the (finite) output space. Let ℱ \mathcal{F} be the class of all functions mapping 𝒳 ↦ P ⁡ ( 𝒴 ) \mathcal{X}\mapsto P(\mathcal{Y}) (where we use the notation P ⁡ ( 𝒴 ) P(\mathcal{Y}) to denote probability distributions over 𝒴 \mathcal{Y} ). That is, the functions in ℱ \mathcal{F} map inputs in 𝒳 \mathcal{X} to probability distributions over 𝒴 \mathcal{Y} , so each function f ∈ ℱ f\in\mathcal{F} defines a conditional probability distribution of y ∈ 𝒴 y\in\mathcal{Y} given x ∈ 𝒳 x\in\mathcal{X} . We denote this distribution by f ⁡ ( y | x ) f(y|x) .

Fix some input distribution p p over 𝒳 \mathcal{X} such that p p has full support (namely, for every x ∈ 𝒳 x\in\mathcal{X} we have p ⁡ ( x ) > 0 p(x)>0 ). Throughout the paper, we assume that our data is labeled by k k experts, denoted f 1 , … , f k ∈ ℱ f_{1},\dots,f_{k}\in\mathcal{F} . Namely, we assume that the inputs are sampled from the input distribution p p and then each input x ∈ 𝒳 x\in\mathcal{X} is labeled by some expert chosen uniformly at random 3 3 3 Equivalently, we can assume that each example is labeled by all experts. . This process induces a joint probability distribution over 𝒳 × 𝒴 \mathcal{X}\times\mathcal{Y} , which we denote by D \dist . Specifically, D ( x , y ) = p ⁡ ( x ) ​ f ¯ ​ ( y | x ) \dist(x,y)=p(x)\overline{f}(y|x) where f ¯ \overline{f} is the mixture of the expert distributions, namely f ¯ ​ ( y | x ) = 1 k ​ ∑ i = 1 k f i ​ ( y | x ) \overline{f}(y|x)=\frac{1}{k}\sum_{i=1}^{k}f_{i}(y|x) (1) We measure the quality of some prediction function f ∈ ℱ f\in\mathcal{F} using a reward assigned to each input-output pair. Namely, we define a reward function r : 𝒳 × 𝒴 → ℝ r:\mathcal{X}\times\mathcal{Y}\to\mathbb{R} , s.t. for all x x , the function r ⁡ ( x , ⋅ ) r(x,\cdot) is not constant (i.e., for every input x x not all outputs have the same reward). We choose some test distribution p test p_{\mathrm{test}} over 𝒳 \mathcal{X} , and for some f ∈ ℱ f\in\mathcal{F} define the average reward of f f over p test p_{\mathrm{test}} by: R p test ( f ) = 𝔼 x ∼ p test [ r x ( f ) ] , where r x ( f ) = 𝔼 y ∼ f ( ⋅ | x ) [ r ( x , y ) ] R_{p_{\mathrm{test}}}(f)=\mathbb{E}_{x\sim p_{\mathrm{test}}}\left[r_{x}(f)\right],~~~\mathrm{where}~~r_{x}(f)=\mathbb{E}_{y\sim f(\cdot|x)}\left[r(x,y)\right] (2) A learner has access to the distribution D \dist , and needs to find a function that minimizes the cross-entropy loss over D \dist . Namely, the learner chooses some function f ^ ∈ ℱ \hat{f}\in\mathcal{F} s.t. f ^ = arg ⁡ min f ∈ ℱ ​ 𝔼 x ∼ p ​ [ H ⁡ ( f ¯ , f ) ] \hat{f}=\arg\min_{f\in\mathcal{F}}\mathbb{E}_{x\sim p}\left[H(\overline{f},f)\right] where H H is the cross-entropy function.

###### Definition 1 .

We define “transcendence” to be a setting of f 1 , … , f k ∈ ℱ f_{1},\dots,f_{k}\in\mathcal{F} and p ∈ P ⁡ ( 𝒳 ) p\in P(\mathcal{X}) where: R p test ​ ( f ^ ) > max i ∈ [ k ] ⁡ R p test ​ ( f i ) R_{p_{\mathrm{test}}}(\hat{f})>\max_{i\in[k]}R_{p_{\mathrm{test}}}(f_{i}) (3)

In other words, transcendence describes cases where the learned predictor performs better (achieves better reward) than the best expert generating the data. Note that we are focusing on an idealized setting, where the learner has access to infinite amount of data from the distribution D \dist , and can arbitrarily choose any function to fit the distribution (not limited to a particular choice of architecture or optimization constraints). As we will show, even in this idealized setting, transcendence can be impossible to achieve without further modifying the distribution.

###### Remark 1 .

We have made various simplifying assumptions when introducing our setting. For example, we assume that all experts share the same input distribution, we assume that all inputs have non-zero probability under the training distribution p p , and we assume the experts are sampled uniformly at random. We leave a complete analysis of a more general setting to future work, and discuss this point further in section 6 .

## 3 Conditions for Transcendence

In this section we analyze the necessary and sufficient conditions for transcendence in our setting. We begin by showing that low-temperature sampling is necessary for transcendence in our specific setting. Then, we analyze specific sufficient conditions for transcendence, both in the case where the data is generated by a single expert and when the data is generated by multiple experts. We defer all proofs to Appendix A .

### 3.1 Low-Temperature Sampling is Necessary for Transcendence

Observe that by definition of f ^ \hat{f} , and using standard properties of the cross-entropy loss, we get that f ^ = f ¯ \hat{f}=\overline{f} , as defined in Eq. ( 1 ). Therefore, the conditional probability distribution generated by f ^ \hat{f} is simply an average of the distributions generated by the expert. Since the reward is a linear function of these distributions, we get that f ^ \hat{f} never achieves transcendence:

###### Proposition 1 .

For all choice of f 1 , … , f k f_{1},\dots,f_{k} and p test p_{\mathrm{test}} , there exists some f i f_{i} s.t. R p test ​ ( f i ) ≥ R p test ​ ( f ^ ) R_{p_{\mathrm{test}}}(f_{i})\geq R_{p_{\mathrm{test}}}(\hat{f}) .

Note that in our setting, we assume that all experts are sampled uniformly for a given input x x . If instead this assumption is removed, then it may be possible to achieve transcendence with a bayesian weighting. We leave this analysis for future work.

### 3.2 Transcendence with Low-Temperature Sampling

Now, we consider a temperature sampling scheme over the learned function f ^ \hat{f} . Namely, for some temperature τ > 0 \tau>0 , and some probability distribution q ∈ P ⁡ ( 𝒴 ) q\in P(\mathcal{Y}) , denote the softmax operator with temperature τ \tau by softmax ⁡ ( q , τ ) ∈ P ⁡ ( 𝒴 ) \mathrm{softmax}(q;\tau)\in P(\mathcal{Y}) s.t. softmax ​ ( q , τ ) y = exp ⁡ ( q y / τ ) ∑ y ′ ∈ 𝒴 exp ⁡ ( q y ′ / τ ) \mathrm{softmax}(q;\tau)_{y}=\dfrac{\exp(q_{y}/\tau)}{\sum_{y^{\prime}\in\mathcal{Y}}\exp(q_{y^{\prime}}/\tau)} . Additionally, we define arg ​ max ⁡ ( q ) ∈ P ⁡ ( 𝒴 ) \argmax(q)\in P(\mathcal{Y}) to be the uniform distribution over the maximal values of q q , namely arg ​ max ⁡ ( q ) = 1 / | Y q | \argmax(q)=1/{\left\lvert Y_{q}\right\rvert} if y ∈ Y q y\in Y_{q} and 0 if y ∉ Y q y\notin Y_{q} , where Y q = { y ∈ 𝒴 : q y = max ⁡ ( q ) } Y_{q}=\{y\in\mathcal{Y}:q_{y}=\max(q)\} . Now, define f ^ τ \hat{f}_{\tau} to be the temperature sampling of f ^ \hat{f} , i.e. f ^ τ ( ⋅ | x ) = softmax ( f ^ ( ⋅ | x ) ; τ ) \hat{f}_{\tau}(\cdot|x)=\mathrm{softmax}(\hat{f}(\cdot|x);\tau) and f ^ max \hat{f}_{\max} the arg-max “sampling” of f ^ \hat{f} , i.e. f ^ max ( ⋅ | x ) = arg ​ max ( f ^ ( ⋅ | x ) ) \hat{f}_{\max}(\cdot|x)=\argmax(\hat{f}(\cdot|x)) . We now show that if the arg-max predictor f ^ max \hat{f}_{\max} is better than the best expert, then transcendence is possible with low-temperature sampling.

###### Proposition 2 .

R p test ​ ( f ^ max ) > max i ∈ [ k ] ⁡ R p test ​ ( f i ) R_{p_{\mathrm{test}}}(\hat{f}_{\max})>\max_{i\in[k]}R_{p_{\mathrm{test}}}(f_{i}) if and only if there exists some temperature τ ∈ ( 0 , 1 ) \tau\in(0,1) s.t. for all 0 ≤ τ ′ ≤ τ 0\leq\tau^{\prime}\leq\tau , it holds that R p test ​ ( f ^ τ ′ ) > max i ∈ [ k ] ⁡ R p test ​ ( f i ) . R_{p_{\mathrm{test}}}(\hat{f}_{\tau^{\prime}})>\max_{i\in[k]}R_{p_{\mathrm{test}}}(f_{i}).

The above shows that, even though transcendence cannot be achieved when directly modeling the distribution, it can be achieved by temperature sampling, assuming that the arg-max predictor achieves higher reward compared to all experts. In other words, we make the subtle connection here that low-temperature sampling can be thought of as performning majority vote [ 1 , 6 ] between the experts. Please see Appendix A for a formal proof of this connection. When the experts put non-negligible mass onto the best actions, the resulting majority vote may find the best action [ 9 ] , which improves performance compared to individual experts (i.e., “wisdom of the crowd”) and thus achieve transcendence.

### 3.3 Denoising a Single Expert

We now turn to study particular cases where low-temperature sampling can lead to transcendence. The most simple case is of a single expert that outputs a correct but noisy prediction. Denote by f ∗ f^{*} the optimal expert, s.t. for all x x we have f ∗ ​ ( y | x ) = δ ⁡ ( y ∈ Y x ∗ ) | Y x ∗ | f^{*}(y|x)=\dfrac{\delta(y\in Y^{*}_{x})}{\lvert Y^{*}_{x}\rvert} , where Y x ∗ = { y ∈ 𝒴 : y = max y ′ ⁡ r ⁡ ( x , y ′ ) } Y^{*}_{x}=\{y\in\mathcal{Y}:y=\max_{y^{\prime}}r(x,y^{\prime})\} and δ ⁡ ( condition ) \delta(\text{condition}) is 1 if the condition is true and 0 otherwise. Now, for some ρ ∈ ( 0 , 1 ) \rho\in(0,1) , let f ρ f_{\rho} be a “noisy” expert, s.t., for all x x , with probability ρ \rho chooses a random output, and with probability 1 − ρ 1-\rho chooses an output according to the optimal expert f ∗ ( ⋅ | x ) f^{*}(\cdot|x) , namely f ρ ​ ( y | x ) = ρ / | 𝒴 | + ( 1 − ρ ) ​ f ∗ ​ ( y | x ) f_{\rho}(y|x)=\rho/\left\lvert\mathcal{Y}\right\rvert+(1-\rho)f^{*}(y|x) . We show that transcendence is achieved with low-temperature sampling for data generated by f ρ f_{\rho} :

###### Proposition 3 .

Assume the data is generated by a single expert f ρ f_{\rho} . Then, there exists some temperature τ ∈ ( 0 , 1 ) \tau\in(0,1) s.t. for all τ ′ ≤ τ \tau^{\prime}\leq\tau , the predictor f ^ τ ′ \hat{f}_{\tau^{\prime}} achieves “transcendence”.

### 3.4 Transcendence from Multiple Experts