Learning to Compress Prompts with Gist Tokens — part 3/3

Cleaned-text code points [31625, 40309). Bibliography omitted here; source.md includes it.

For LLaMA-7B, we used a maximum sequence length of 512 tokens during training and evaluation, except with the Human validation split, where the maximum length was increased to 768 (the Human instructions are longer). Examples longer than this length are truncated from the end. For FLAN-T5-XXL, we set a maximum input length (task t t + input x x ) of 128 and a maximum output length of 256, except again for the Human split, where the maximum input and output lengths were both set to 384. For both models, we set a maximum generation length of 512 tokens. These lengths were chosen such that < 1 % <1\% of examples across the board were truncated during training and evaluation for both models.

#### Training.

Full hyperparameters for training runs are located in Table A.1 . These parameters were adapted from previous published work finetuning LLAMA/FLAN-T5. For LLaMA-7B, parameters are identical to those used in training Alpaca Taori et al. [31] . For FLAN-T5-XXL, parameters are identical to those used in training T k k - Instruct [ 37 ] , except with a 5e-5 learning rate, as used in the T k k - Instruct GitHub repository, 4 4 4 https://github.com/yizhongw/Tk-Instruct/blob/1ab6fad/scripts/train_tk_instruct.sh rather than the 1e-5 learning rate in the paper.

LLaMA-7B was trained for 3000 steps, while FLAN-T5-XXL was trained for 16000 steps. Since there are about 130k examples in Alpaca+, given the batch sizes in Table A.1 this corresponds to about ~3 epochs and ~2 epochs of training, respsectively. These numbers, again, are identical to Taori et al. [31] and Wang et al. [36] . We note that the training time is relatively flexible; for example, we did not see substantial gains training beyond 1 epoch for FLAN-T5-XXL.

#### Evaluation.

During evaluation and benchmarking, we simply greedily decoded the most likely sequence. We saw limited gains from beam search with beam size B = 4 B=4 .

#### Compute.

Experiments were run on a cluster machine with 4xA100-SXM4-80GB NVIDIA GPUs, 480GB RAM, and 16 CPUs, using PyTorch 2.0 [ 24 ] , Hugging Face Transformers [ 41 ] , and DeepSpeed [ 29 ] . Training runs take about ~7 hours to complete for LLaMA-7B and ~25 hours for FLAN-T5-XXL. Benchmarking results were obtained on the same machine, but using just 1 of the A100 GPUs.

## Appendix C Details of ChatGPT Evaluation

We used the ChatGPT API, specifically the chatgpt -3.5- turbo engine, to run our ChatGPT evaluation experiments over a period of 2 weeks between March 27 and April 7, 2023.

The full prompt given to ChatGPT is located in Listing A.1 , and contains 4 examples of desired output from ChatGPT, including preferring factually accurate responses (Example 1), preferring responses that follow the instruction, even if imperfect (Examples 2 and 3), and examples of models being equally wrong (Examples 4). For the two models under comparison, we randomized the order of presentation of each model as either Assistant A or Assistant B, to avoid order effects.

ChatGPT was instructed to only respond in JSON format, outputting first a reason key followed by a choice key, to encourage chain-of-thought reasoning [ 39 ] . On rare occasions ( < 0.25 % <0.25\% of the time), ChatGPT would output a response that did not conform to the requested JSON format (e.g. it would just give an unstructured paragraph). In these cases we manually went through and converted these responses to JSON, without altering ChatGPT’s reasoning.

In total, we collected ~22.5k judgments from ChatGPT for an estimated cost of $29.28. The full outputs for each model across the Alpaca+ validation splits, as well as ChatGPT’s responses and choices, are available in the code link above.

## Appendix D Additional Human Evaluation Details and Results

### D.1 Experimental Details

For each of the 100 examples randomly selected from the Human validation split, we recruited 3 US or UK-based, English-fluent annotators from Prolific, an online crowdsourcing platform. Experiments were IRB approved under a generic human experiments IRB given to the authors.

The annotation interface given to Prolific crowdworkers is located in Figure A.2 . To verify task comprehension, participants were shown two simple examples before the main body of the task (Figure A.3 ), and were required to answer correctly before proceeding. We compensated participants USD $14.35/hour for an estimated cost (including Prolific fees) of USD $141.64.

### D.2 Additional Results

See Table A.2 for a breakdown of Cohen’s κ \kappa between human annotators and ChatGPT. We used a weighted version of Cohen’s κ \kappa with linear weights, since the response scale is ordinal (e.g. “tie” is a closer judgment to “pos control win” than “gist win”).

## Appendix E Exact Match Results

See Figure A.4 for a plot of exact match rates for the gist and positive control models (as measured by exact string match).

## Appendix F Additional FLOPs details

The FLOPs required for a Transformer formward pass with varying KV cache lengths can be estimated by modifying existing equations to account for self-attention back to the KV cache. As an example, we modify the FLOPs equations used for computing FLOPs in the Chinchilla paper (Appendix F in [ 14 ] ). Let seq_len_with_past = seq_len + kv_cache_len. Then the modified Transformer FLOPs equations are:

#### Embeddings

• 2 × \times seq_len × \times vocab_size × \times d_model

#### Attention (Single Layer)

• Key, query, and value projections : 2 × \times 3 × \times seq_len × \times d_model × \times (key_size × \times num_heads)

• Key and query logits : 2 × \times seq_len × \times seq_len_with_past × \times (key_size × \times num_heads)

• Softmax : 3 × \times num_heads × \times seq_len × \times seq_len_with_past

• Softmax @ query reductions : 2 × \times seq_len × \times seq_len_with_past × \times (key_size × \times num_heads)

• Final linear : 2 × \times seq_len × \times (key_size × \times num_heads) × \times d_model

#### Dense Block

• 2 × \times seq_len × \times (d_model × \times ffw_size + d_model × \times ffw_size)

#### Final Logits

• 2 × \times seq_len × \times d_model × \times vocab_size

#### Total Forward Pass FLOPs

• embeddings + num_layers × \times (attention_single_layer + dense_block) + final_logits

It can be seen that only 3 operations in each attention layer depend on the KV cache size, and they take up a relatively insignificant amount of FLOPs. As an illustrative example, Figure A.5 shows the relative FLOPs contributions within a single layer of attention for LLaMA-7B, assuming a 2000-length KV cache and a single input token. Operations dependent on the KV cache constitute at most ~10% of the total attention layer FLOPs; the rest are used in KQV projections and dense layers for processing the single new input token.

Given a KV cache compression rate of 26, as observed in our Human validation split, the Chinchilla equations predict a relative improvement of Gist caching of 0.12%. This is extremely close to the 0.11% improvement actually observed in Table 3 . These results show that optimizing the KV cache size does not actually lead to huge compute speedups during Transformer inference, at least for relatively small prompt lengths. Nevertheless, there are clear memory and storage benefits to be gained from prompt compression, as discussed in Section 6 .

## Appendix G Additional Examples of Model Outputs

See Listing A.2 for additional examples of instructions, inputs, model completions (for LLaMA-7B and FLAN-T5-XXL across Pos, Gist, TF-IDF, and Neg conditions), human judgments, and ChatGPT judgments (+reasons). Examples are not cherry-picked; we enumerate the first 10 examples of the Alpaca+ Human validation split.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .