The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

arxiv:1803.03635

Original source · Paper JSON

Data, training, optimization · 2018-03-09

Cites (0) · Cited by (6)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Abstract
• 1 Introduction
• The Lottery Ticket Hypothesis .
• 2 Winning Tickets in Fully-Connected Networks
• 3 Winning Tickets in Convolutional Networks
• 4 VGG and Resnet for CIFAR10
• 5 Discussion
• 6 Limitations and Future Work
• 7 Related Work
• Appendix A Acknowledgments
• Appendix B Iterative Pruning Strategies
• Strategy 1: Iterative pruning with resetting.
• Strategy 2: Iterative pruning with continued training.
• Appendix C Early Stopping Criterion
• Appendix D Training Accuracy for Lottery Ticket Experiments
• Appendix E Comparing Random Reinitialization and Random Sparsity
• Appendix F Examining Winning Tickets
• F.1 Winning Ticket Initialization (Adam)
• F.2 Winning Ticket Initializations (SGD)
• F.3 Reinitializing from Winning Ticket Initializations
• F.4 Pruning at Iteration 0
• F.5 Comparing Initial and Final Weights in Winning Tickets
• F.6 Winning Ticket Connectivity
• F.7 Adding Noise to Winning Tickets
• Appendix G Hyperparameter Exploration for Fully-Connected Networks
• G.1 Experimental Methodology
• G.2 Learning Rate
• G.3 Other Optimization Algorithms
• G.3.1 SGD
• G.3.2 Momentum
• G.4 Iterative Pruning Rate
• G.5 Initialization Distribution
• G.6 Network Size
• Appendix H Hyperparameter Exploration for Convolutional Networks
• H.1 Experimental Methodology
• H.2 Learning Rate
• H.3 Other Optimization Algorithms
• H.3.1 SGD
• H.3.2 Momentum
• H.4 Iterative Pruning Rate
• H.5 Learning Rates (Dropout)
• H.6 Pruning Convolutions vs. Pruning Fully-Connected Layers
• Appendix I Hyperparameter Exploration for VGG-19 and Resnet-18 on CIFAR10
• I.1 Global Pruning
• I.2 VGG-19 Details
• I.3 Resnet-18 Details
• I.4 Learning Rate
• I.5 Warmup Iteration
• Resnet-18.
• VGG-19.
• Instructions for reporting errors