Transcendence: Generative Models Can Outperform The Experts That Train Them — part 4/4

Cleaned-text code points [46435, 49235). Bibliography omitted here; source.md includes it.

Interestingly, an analogue to low-temperature sampling also has been noticed and exploited by Reinforcement Learning practitioners in the context of off-policy learning , where a different exploration policy π E \pi_{E} is used than the final learned target policy π T \pi_{T} . Oftentimes π T \pi_{T} will just be set to a greedy version of π E \pi_{E} [ 21 ] , such as choosing π T \pi_{T} to take the arg ​ max \argmax action of π E \pi_{E} , which we note is directly equivalent to setting temperature to 0.

## Appendix E Training Details

We give a full list of the hyperparameters we used for training here. Note that we largely follow the same hyperparameter set as [ 37 ] , but lower the batch size to 125 ​ K 125K as we found training to still be stable ta this level. We also release our code openly to support further research into transcendence, which was built off the wonderful work done by Karvonen [12] and Karpathy [11] .

## Appendix F Compute Resources

We train all of our models on the Nvidia H100 80GB GPU. To train one of our models takes around 6 to 12 hours.

## Appendix G Full t-SNE

We visualize the full t-SNE here, coloring by the reward of the game. We see that the model has learned some representation of the reward, with high absolute reward states being more likely to be near each other in the latent space. This also points towards evidence that the model has learned some sort equivariant representation of the player identity, as the region of symmetric high reward states indicate. Note that reward is not directly given to the model during training.

We visualize the same t-SNE, but this time coloring by game length rather than reward. We see that games with high reward tend to be longer, which makes logical sense as the result of the game will tend to be clearer as the game proogresses.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .