Physics of Language Models: Part 3.1, Knowledge Storage and Extraction — part 6/6

Cleaned-text code points [78898, 79254). Bibliography omitted here; source.md includes it.

For Q-probing, we use the AdamW optimizer with ε = 10 − 6 \varepsilon=10^{-6} , weight decay 0.3, an initial learning rate of 0.001 0.001 , no warmup, linear learning rate decay (down to 0), a batch size of 200, and 30000 training steps. This is identical to the procedure outlined in Appendix F .

All of our results were presented in the same Figure 9 .