Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs — part 3/3

Cleaned-text code points [31804, 34630). Bibliography omitted here; source.md includes it.

(B) “Alternating” Another option is to alternate attention patterns with the following procedure: 1. Prefill the first instruction and generate the initial KV cache by merging causal and bidirectional features.

2. Generate output token by token, appending the KV of each output to the cache (the output KV uses only causal features).

3. Prefill the next instruction with both attention masks and append its KV to the cache.

4. Repeat from step 2.

### A.9 Chain of Thought Data Generation

For each train set of datasets - PIQA, ARC-Challenge, CSQA, GSM8K - we have distilled chain-of-thought completions with GPT-4.1 model.

The completions were asked to be in a specific format allowing to extract the steps, and final answer separately. We have used the following prompt templates:

Here we provide the example completions:

### A.10 Prompt Templates

Templates used to format instruction-answer pairs for a given dataset, for both training and evaluation. In all cases there is a space character at the beginning of the answer part.

### A.11 Results with Standard Deviation

Tables with complete results averaged over 3 seeds, includes standard deviation.

### A.12 Example Attention Matrices

Here we show pairs of matrices with causal and bidirectional attention scores of Bituned Gemma-2B for the first two GSM8K samples, with scores averaged over layers and heads. Source tokens are represented with the vertical axis, while target tokens (for which the attention is paid to) with the horizontal one - e.g. the first column represents attention paid to the first (BOS) token, by each other token. Darker color represents higher attention score.

### A.13 GSM8K Samples

Samples of responses to the first 20 questions from GSM8K benchmark. Shown for Llama3-8B intruction-tuned on UltraFeedback dataset - for standard finetuning with LoRA and Bitune.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .