Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs

arxiv:2405.14862

Original source · Paper JSON

LLMs: architectures, context, training · 2024-05-23

Cites (1) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs
• Abstract
• 1 Introduction
• 2 Bidirectional Instruction-Tuning
• Two Sets of Features.
• Parameter Efficient Fine-tuning
• 3 Experiments
• 3.1 Instruction-Tuning
• Results.
• 3.2 Downstream Task Training
• Results.
• 3.3 Chain-of-Thought Reasoning
• Results.
• 3.4 Inference Speed Comparison
• 3.5 Ablations
• Component Removal
• Different PEFT Methods
• Full Finetuning
• Attention Mask of Second Pass
• 4 Related Work
• Bitune vs. Prefix-LM.
• 5 Conclusion
• Limitations
• Appendix A Appendix
• A.1 Pseudocode for Bitune Training Step
• A.2 Hyperparameters
• A.3 Datasets
• A.4 Training Speed & Memory Usage
• A.5 Initialization of Mixing Coefficient
• A.6 Instruction-tuning on Alpaca Dataset
• A.7 Finetuning Larger Model on GSM8K
• A.8 Multi-Turn Setting
• A.9 Chain of Thought Data Generation
• A.10 Prompt Templates
• A.11 Results with Standard Deviation
• A.12 Example Attention Matrices
• A.13 GSM8K Samples
• Instructions for reporting errors