Energy Transformer

arxiv:2302.07253

Original source · Paper JSON

LLMs: architectures, context, training · 2023-02-14

Cites (0) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Energy Transformer
• Abstract
• 1 Introduction
• 2 Energy Transformer Block
• Layer Norm
• Multi-Head Energy Attention
• Hopfield Network Module
• Dynamics of Token Updates
• Relationship to Modern Hopfield Networks and Conventional Attention
• 3 Qualitative Inspection of the ET framework on ImageNet
• 4 Graph Anomaly Detection
• 4.1 Experimental Evaluation
• 5 Graph Classification with ET
• 5.1 Experimental Evaluation
• 6 Discussion and Conclusions
• Appendix A Notations Used in the Main Text and Appendices
• Appendix B Details of Training on ImageNet
• B.1 Exploring the Hopfield Memories
• B.2 Positional Bias Correlations
• Appendix C Details of ET training on Anomaly Detection Task
• C.1 Detailed Model Structure for the Graph Anomaly Detection
• C.2 Experimental Details
• Appendix D Graph Classification with ET
• D.1 Details of Graph Classification ET Model
• D.2 Experimental Evaluation
• D.3 Experimental Details
• Appendix E ET for heterogeneous Graph
• Appendix F Ablation Study for Attention and Hopfield Network Modules
• F.1 On Graphs
• F.2 On Images
• Appendix G Parameter comparison
• Appendix H Formal Algorithm for Training and Inference of ET
• Instructions for reporting errors