Transcendence: Generative Models Can Outperform The Experts That Train Them — part 2/4
Cleaned-text code points [15039, 30828). Bibliography omitted here; source.md includes it.
Next, we consider the case where the dataset is generated by multiple experts that complement each other in terms of their ability to correctly predict the best output. For example, consider the case where the input space is partitioned into k k disjoint subsets, 𝒳 = 𝒳 1 ∪ ˙ … ∪ ˙ 𝒳 k \mathcal{X}=\mathcal{X}_{1}\dot{\cup}\dots\dot{\cup}\mathcal{X}_{k} , s.t. the i i -th expert performs well on the subset 𝒳 i \mathcal{X}_{i} , but behaves randomly on other subsets. Namely, assume the expert f i f_{i} behaves as follows: f i ( y | x ) = ( δ ( y ∈ Y x ⋆ ) δ ( x ∈ 𝒳 i ) | Y x ⋆ | + δ ( x ∉ 𝒳 i ) | 𝒴 | ) f_{i}(y|x)=\biggl(\frac{\delta(y\in Y^{\star}_{x})\delta(x\in\mathcal{X}_{i})}{|Y^{\star}_{x}|}+\frac{\delta(x\notin\mathcal{X}_{i})}{|\mathcal{Y}|}\biggr) where Y x ∗ Y_{x}^{*} is as previously defined and δ ( condition ) \delta(\text{condition}) is 1 if the condition is true and 0 otherwise. We show that, assuming that the test distribution p test p_{\mathrm{test}} is not concentrated on a single subset 𝒳 i \mathcal{X}_{i} , we achieve transcendence with low-temperature sampling:
###### Proposition 4 .
Let p test p_{\mathrm{test}} be some distribution s.t. there are at least two subsets 𝒳 i ≠ 𝒳 j \mathcal{X}_{i}\neq\mathcal{X}_{j} s.t. p test ( 𝒳 i ) , p test ( 𝒳 j ) > 0 p_{\mathrm{test}}(\mathcal{X}_{i}),p_{\mathrm{test}}(\mathcal{X}_{j})>0 . Then, if the data is generated by f 1 , … , f k f_{1},\dots,f_{k} , there exists some temperature τ ∈ ( 0 , 1 ) \tau\in(0,1) s.t. for all τ ′ ≤ τ \tau^{\prime}\leq\tau , the predictor f ^ τ ′ \hat{f}_{\tau^{\prime}} achieves “transcendence”.
In order to build intuition for Proposition 4 , see Appendix C for an intuitive diagram.
## 4 Experiments
To evaluate the predictive power of our impossibility result of transcendence with no temperature sampling ( Proposition 1 ) as well as our result of transcendence from multiple experts with low temperature sampling ( Proposition 2 ), we turn to modeling and training chess players. Chess stands out as an attractive option for several reasons. Chess is a well-understood domain and more constrained than other settings such as natural language generation, lending to easier and stronger analysis. Evaluation of skill in chess is also natural and well-studied, with several rigorous statistical rating systems available. In this paper, we use the Glicko-2 rating system [ 7 ] , which is also adopted by https://lichess.org , the free and open-source online chess server from which we source our dataset.
### 4.1 Experimental Setup
#### Training Details.
We trained several 50 50 M parameter autoregressive transformer decoders following best practices from modern large model training, including a cosine learning rate schedule and similar batch size-learning rate ratios as prescribed by the OPT-175B team [ 37 ] . Our dataset consists of human chess games from the lichess.org open source database from January 2023 to October 2023. In total, this dataset contains approximately one billion games. In this setting, an expert is a specific individual player. To test for transcendence, we truncate this dataset by a maximum rating, so that during training a model only sees data up to a given rating. We train our model on the next-token prediction objective, and represent our chess games as Portable Game Notation (PGN) strings, such as 1.e4 e5 2.Nf3 Nc6 3.Bb5... 1/2-1/2 . Note that we do not give any rating or reward information during training—the only input the model sees are the moves and the outcome of the game. We tokenize our dataset at the 32 32 -symbol character level. (For further details, see Appendix E .) Our model plays chess “blind”—without direct access to the board state—and, furthermore, is never explicitly given the rules of the game: at no point is play constrained to valid outputs for a given piece or board state. Nontrivial chess skill is therefore not straightforward to acquire, and if not for the surprising capabilities of modern large transformers, one might imagine such a model would fail to learn even the basic rules of playing chess. This blindfolded setting has also been studied by prior work [ 23 , 30 ] , as discussed further in section 5 .
One gap between our theory and practice is that in our theory, we assume that each expert is defined over the entire input space 𝒳 \mathcal{X} . However, in the chess setting such full coverage is extremely unlikely to be the case after around move 15 15 , as there are more unique chess games than atoms in the universe due to the high branching factor of the game tree. To address this gap, we visualize the latent representation of our model in Figure 3 , where we find the model is able to capture meaningful semantics regarding both the relative advantage of a state, as well as the identity of the black and white player. This visualization illustrates the ability of our model to generalize by compressing games into some shared latent representation, enabling experts to generalize to unseen states, bridging this gap between theory and practice.
#### Evaluation.
We evaluate each model by its Glicko-2 ratings against Stockfish 16.1 [ 29 ] , a popular open-source chess engine. Stockfish uses a traditional minimax search equipped with a bespoke CPU-efficient neural network for evaluation [ 22 ] and α \alpha - β \beta pruning for further efficiency. We evaluate Stockfish at levels 1, 3, and 5 with a 100ms timeout directly on Lichess’ platform against the Maia [ 18 ] 1, 5, and 9 bots (human behavior cloned convolutional networks trained at rating bins 1100-1200, 1500-1600, and 1900-2000, respectively) for several hundred games, obtaining calibrated Glicko-2 ratings for Stockfish specifically on Lichess’ platform ( 1552 ± 45.2 1552\pm 45.2 , 1842 ± 45.2 1842\pm 45.2 , 2142 ± 59 2142\pm 59 for Stockfish Levels 1, 3, and 5, respectively). Next, for evaluating our own models, we then play against Stockfish levels of 1, 3, and 5 for 100 games each, reaching a final rating calculation with 300 games. We then report both the Glicko-2 rating R R as well as rating deviation R D RD of our models, where R ± 2 ∗ R D R\pm 2*RD provides a 95 % 95\% confidence interval. To play against Stockfish, we successively prompt our model with the current game PGN string. Note that our output is entirely unconstrained, and may be either illegal in the current board state or altogether unparsable. If our model fails to generate a valid legal move after 5 samples, we consider it to have lost. After generation, we give the updated board state to Stockfish and pass a new PGN string appended with the prior move of Stockfish back to our model. We repeat this process until the game ends.
### 4.2 Experimental Results
#### Main Result: Low-temperature sampling enables transcendence.
In this section we attempt to answer our primary research question, can low-temperature sampling actually induce transcendence in practice? We test Proposition 2 by evaluating several ChessFormers across different temperature values, from 0.001 0.001 (nearly deterministic), to 1.0 1.0 (original distribution), to 1.5 1.5 (high entropy). In Figure 1 we definitively confirm the existence of transcendence. Our ChessFormer 1000 (where the latter number refers to the maximum rating seen during training) and ChessFormer 1300 models are able to transcend to around 1500 rating at temperature τ \tau equal to 0.001 0.001 . Interestingly, ChessFormer 1500 is unable to transcend at test time, a result we further analyze in Dataset Diversity .
To more deeply understand when and why transcendence occurs, we investigate two questions. (1) How does the reward function defined in Equation 2 shift with respect to low-temperature sampling? (2) Does transcendence rely on dataset diversity, as introduced theoretically in subsection 3.4 ?
#### Lowering temperature increases rewards in expectation on specific states, leading to transcendence over the full game.
When playing chess, a low-skilled player may play reasonably well until they make a significant blunder at a key point in play. If these errors are idiosyncratic, averaging across many experts would have a denoising effect, leaving the best moves with higher probability. Therefore, low-temperature sampling would move probability mass towards better moves in specific play contexts. Without low-temperature sampling, the model would still put probability mass onto blunders. To gain intuition for this idea, we visualize it theoretically in Appendix C and empirically in Figure 2 and Appendix B . This hypothesis motivates our first research question in this section: Does low-temperature sampling improve the expected reward very much for just some specific key game states, or a little for many game states?
To formalize this notion, we first define a “favor” function, which captures the improvement in reward by following some new probability distribution over some baseline probability distribution. Our definition is inspired by the Performance Difference Lemma (PDL) [ 10 ] from Reinforcement Learning (RL), which establishes an equivalence between the change in performance from following some new policy (a probability distribution of actions given a state) over some old policy, and the expected value of the advantage function of the old policy sampled with respect to the new policy. In RL, the advantage function is defined as the difference between the value of taking a single action in a given state versus the expected value of following some policy distribution of actions in that state.
Here, we define the “favor” of f ′ f^{\prime} over f f in x x as the change in the reward function by comparing what f f would have done when following f ′ f^{\prime} for a given input x x :
F ( f ′ , f ; x ) = 𝔼 x ∼ d f ′ , y ∼ f ′ ( ⋅ | x ) [ r ( x , y ) ] − 𝔼 x ∼ d f ′ , y ∼ f ( ⋅ | x ) [ r ( x , y ) ] . F(f^{\prime},f;x)=\mathbb{E}_{x\sim d^{f^{\prime}},y\sim f^{\prime}(\cdot|x)}[r(x,y)]-\mathbb{E}_{x\sim d^{f^{\prime}},y\sim f(\cdot|x)}[r(x,y)]. (4)
Where d f d^{f} refers to the state visitation distribution [ 31 ] when following f f in a sequential setting—informally, this variable can be thought of the distribution of states seen when sampling from f f with a fixed transition function that takes in an input x x , a output y y , and outputs a next input x x . Here, that transition function is given by the rules of chess and the opponent player. Given this favor function, we can now quantitatively explore the effects that lead to transcendence by setting the baseline f f to be the original imitation-learned probability distribution (temperature τ = 1 \tau=1 ), and f ′ f^{\prime} as a low-temperature intervention on f f (e.g. temperature τ = 0 \tau=0 ). We can empirically calculate the reward by using the evaluation function [ 22 ] of Stockfish, an expert neural reward function that Stockfish uses to calculate its next move. This reward function is a neural network trained to predict the probability of winning through a sigmoid on a linear combination of handcrafted expert heuristics, such as amount of material versus opponent material, and number of moves to a potential checkmate.
In Figure 4 , we find that lowering the temperature has the effect of skewing the expected reward distribution to the right, especially for the green τ = 0.001 \tau=0.001 distribution. This result implies that the model does not improve the expected reward by a small amount for many game states, but rather improves the expected reward by a relatively large amount for a few game states. Thus, τ = 0.001 \tau=0.001 improves the expected reward (probability of winning) by an average of 2.15 ± 0.17 % \mathbf{2.15\pm 0.17\%} , but for some states, this expected improvement is over 5%. Note that the original temperature expected reward can be thought of as a Dirac distribution centered at 0 0 . The above finding answers our research question in this section: Low-temperature sampling is able improves the expected reward by relatively large amounts for some specific game states, which is likely why the ChessFormer 1000 1000 and 1300 1300 model was able to achieve transcendence.
In Table 1 , we present the statistics of the favor function for different temperature values. From this table, we observe that as the temperature decreases, the top- k k accuracies monotonically increase, suggesting that the model becomes more consistent in selecting good moves. We also observe that although the model improves as temperature decreases, the probability of winning is still below 50 % 50\% , meaning our model should tend to lose more games than it wins against Stockfish 1 1 . This result matches with our results in Figure 1 , as the rating of Stockfish 1 1 is also higher than the reported rating for τ = 0.001 \tau=0.001 ( 1550 1550 for Stockfish 1 vs ∼ 1450 \sim 1450 for Chessformer 1000 1000 ). Overall, the analysis of the advantage statistics provides further evidence for the effectiveness of low-temperature sampling in inducing transcendence in chess models.
#### Dataset diversity is essential for transcendence.
As we note in subsection 3.4 , our theory requires dataset diversity as a necessary condition for enabling transcendence. Importantly, we find in Figure 1 that not all models are able to transcend. Unlike ChessFormer 1000 or 1300, the Chessformer 1500 fails to transcend. We hypothesize that this results is due to the fact that in the band of ratings from 1000 1000 to 1500 1500 , diversity does not significantly increase. If so, a 1000 1000 rated player can be thought of as a noisy 1500 1500 rated player, but a 1500 1500 rated player cannot be thought of as a noisy 2000 2000 rated player. In this section we ask the following research question: Is diversity in data required for enabling transcendence?
In Figure 5 , we explore this research question by quantifying dataset diversity through the normalized entropy on the action distribution ℋ f ( Y | X ) = 𝔼 y ∼ f ( y | x = X ) [ − log 2 f ( y | x = X ) ] / log 2 | 𝒴 | . \mathcal{H}_{f}(Y|X)={\mathbb{E}_{y\sim f(y|x=X)}[-\log_{2}f(y|x=X)]}/{\log_{2}|\mathcal{Y}|}. To gain intuition for this metric, imagine the action distribution of moves taken for any given state. Entropy will be higher for more uniform action distributions, and lower for more deterministic, peaked action distributions. The average entropy of these action distributions can therefore serve as a measurement of the diversity of the dataset. We normalize this entropy to the range [ 0 , 1 ] [0,1] by dividing by the binary log of the number of legal moves: log 2 | 𝒴 | \log_{2}|\mathcal{Y}| .
Importantly, we cannot calculate this normalized entropy for every state, as most states after move 16 16 in the midgame and before the engame are unique within the dataset and we therefore observe just a single action for thus states. Therefore our metric is limited in that it only considers opening moves, the beginning of the midgame, and the endgame. We consider only common states with greater than 100 100 actions by sampling 1,000,000 1,000,000 games from each dataset. The average entropy confirm our hypothesis: The < 1500 <1500 cut off dataset has on average less diversity than the < 1300 <1300 dataset, which has is again less than the < 1000 <1000 dataset. This result suggests that Chessformer 1500 1500 likely is not transcendent due to a lack of diversity in its dataset. If the entropy instead stayed constant for each dataset, it would imply that each had a similar level of diversity. In such a case, we would expect that ChessFormer 1500 1500 likely would also transcend. Instead, as predicted, it is likely not transcendent due to a lack of diversity.
### 4.3 Additional Settings