Pre-trained Large Language Models Use Fourier Features to Compute Addition — part 4/4

Cleaned-text code points [47089, 51451). Bibliography omitted here; source.md includes it.

We finetune GPT-2-XL on the “language-math-dataset” with 50 50 epochs and a batch size of 16 16 . The dataset consists of 27,400 27,400 training samples, 3,420 3,420 validation samples, and 3,420 3,420 test samples. We use the AdamW optimizer, scheduling the learning rate linearly from 1 × 10 − 5 1\times 10^{-5} to 0 0 without warmup.

#### Train GPT-2-XL from scratch

We train GPT-2-XL on the “language-math-dataset” from scratch with 500 500 epochs and a batch size of 16 16 . The dataset consists of 27,400 27,400 training samples, 3,420 3,420 validation samples, and 3,420 3,420 test samples. We use the AdamW optimizer, scheduling the learning rate linearly from 1 × 10 − 4 1\times 10^{-4} to 0 0 without warmup.

#### Train GPT-2 from scratch

For both with pre-trained token embedding and without token embedding, we train GPT-2 on the “language-math-dataset” with 700 700 epochs and a batch size of 16 16 . The dataset consists of 27,400 27,400 training samples, 3,420 3,420 validation samples, and 3,420 3,420 test samples. We use the AdamW optimizer, scheduling the learning rate linearly from 5 × 10 − 5 5\times 10^{-5} to 0 0 without warmup. In Figure 7 b, we train the model with five different seeds and plot the mean and deviation for them.

#### Create the addition dataset in main content

We consider numbers in base 10 10 up to a maximum value of 260 260 . For each pair of numbers between 0 0 and 260 260 , we generate various phrasings of addition questions and their corresponding answers. The different phrasings used are: “Total of num1 and num2.”, “Add together num1 and num2.”, “Calculate num1 + num2.”, “What is the sum of num1 and num2?”, and “Put together num1 and num2.”. The dataset is shuffled to ensure randomness and then split into training ( 80 % 80\% ), validation ( 10 % 10\% ), and test ( 10 % 10\% ) sets.

#### Create the addition dataset in Appendix C.3 with different format

We consider numbers in base 10 10 up to a maximum value of 260 260 . We generate all possible pairs of numbers within this range using combinations with replacement. For each pair, we convert the numbers to the specified base and create questions formatted as “num1,num2+” with their corresponding answers. The dataset is then split into training ( 80 % 80\% ), validation ( 10 % 10\% ), and test ( 10 % 10\% ) sets.

#### Experiments Compute Resources

All experiments involving fine-tuning and training from scratch in this paper were conducted on one NVIDIA A6000 GPU with 48GB of video memory. The fine-tuning process required less than 10 hours, while training from scratch took less than 3 days. Other experiments, such as those involving Logit Lens, were completed in less than 1 hour.

#### Licenses for Existing Assets & Open Access to Data and Code.

For the following models, we use the checkpoints provided by Huggingface. For all the trained models, we use default hyperparameters during all the training but with different random seeds. • GPT-2-XL: https://huggingface.co/openai-community/gpt2-xl , Modified MIT License

• GPT-2: https://huggingface.co/openai-community/gpt2 , Modified MIT License

• GPT-J: https://huggingface.co/EleutherAI/gpt-j-6b , Apache-2.0 License

• Phi2: https://huggingface.co/microsoft/phi-2 , MIT License

• GPT-3.5 and GPT-4: https://chatgpt.com/ or https://openai.com/index/openai-api/

• PaLM-2 https://ai.google/discover/palm2/

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .