On the Information Bottleneck Theory of Deep Learning (Saxe et al.) — part 3/6

Cleaned-text code points [29262, 43699). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2018
from a high to a low gradient signal-to-noise ratio (SNR), i.e., the onset of the diffusion phase. The
proposed mechanism behind this diffusion-driven compression is as follows. The authors state that
during the diffusion phase, the stochastic evolution of the weights can be described as a Fokker-Planck
equation under the constraint of small training error. Then, the stationary distribution over weights
for this process will have maximum entropy, again subject to the training error constraint. Finally,
the authors claim that weights drawn from this stationary distribution will maximize the entropy
of inputs given hidden layer activity,H(X|T ), subject to a training error constraint, and that this
training error constraint is equivalent to a constraint on the mutual information I(T ;Y ) for small
training error. Since the entropy of the input, H(X), is ﬁxed, the result of the diffusion dynamics
will be to minimizeI(X;T ) :=H(X)−H(X|T ) for a given value ofI(T ;Y ) reached at the end
of the drift phase.
However, this explanation does not hold up to either theoretical or empirical investigation. Let
us assume that the diffusion phase does drive the distribution of weights to a maximum entropy
distribution subject to a training error constraint. Note that this distribution reﬂects stochasticity
of weights across different training runs. There is no general reason that a given set of weights
sampled from this distribution (i.e., the weight parameters found in one particular training run)
will maximizeH(X|T ), the entropy of inputs given hidden layer activity. In particular, H(X|T )
reﬂects (conditional) uncertainty about inputs drawn from the data-generating distribution, rather
than uncertainty about any kind of distribution across different training runs.
We also show empirically that the stochasticity of the SGD is not necessary for compression. To do
so, we consider two distinct training procedures: ofﬂine stochastic gradient descent (SGD), which
learns from a ﬁxed-size dataset, and updates weights by repeatedly sampling a single example from
the dataset and calculating the gradient of the error with respect to that single sample (the typical
procedure used in practice); and batch gradient descent (BGD), which learns from a ﬁxed-size dataset,
and updates weights using the gradient of the total error across all examples. Batch gradient descent
uses the full training dataset and, crucially, therefore has no randomness or diffusion-like behavior in
its updates.
We trained tanh and ReLU networks with SGD and BGD and compare their information plane
dynamics in Fig. 5 (see Appendix H for a linear network). We ﬁnd largely consistent information
dynamics in both instances, with robust compression in tanh networks for both methods. Thus
randomness in the training process does not appear to contribute substantially to compression of
information about the input. This ﬁnding is consistent with the view presented in Section 2 that
compression arises predominantly from the double saturating nonlinearity.
Finally, we look at the gradient signal-to-noise ratio (SNR) to analyze the relationship between
compression and the transition from high to low gradient SNR. Fig. 20 of Appendix I shows the
gradient SNR over training, which in all cases shows a phase transition during learning. Hence
the gradient SNR transition is a general phenomenon, but is not causally related to compression.
Appendix I offers an extended discussion and shows gradient SNR transitions without compression
on the MNIST dataset and for linear networks.
5 S IMULTANEOUS FITTING AND COMPRESSION
Our ﬁnding that generalization can occur without compression may seem difﬁcult to reconcile with
the intuition that certain tasks involve suppressing irrelevant directions of the input. In the extreme,
if certain inputs contribute nothing but noise, then good generalization requires ignoring them. To
study this, we consider a variant on the linear student-teacher setup of Section 3: we partition the
inputX into a set of task-relevant inputs Xrel and a set of task-irrelevant inputs Xirrel, and alter
the teacher network so that the teacher’s weights to the task-irrelevant inputs are all zero. Hence the
inputsXirrel contribute only noise, while theXrel contain signal. We then calculate the information
plane dynamics for the whole layer, and for the task-relevant and task-irrelevant inputs separately.
Fig. 6 shows information plane dynamics for a deep linear neural network trained using SGD (5
samples/batch) on a task with 30 task-relevant inputs and 70 task-irrelevant inputs. While the overall
dynamics show no compression phase, the information speciﬁcally about the task-irrelevant subspace
does compress over the course of training. This compression process occurs at the same time as
the ﬁtting to the task-relevant information. Thus, when a task requires ignoring some inputs, the
9

Published as a conference paper at ICLR 2018
A B C
Figure 6: Simultaneous ﬁtting and compression. (A) For a task with a large task-irrelevant subspace
in the input, a linear network shows no overall compression of information about the input. (B)
The information with the task-relevant subspace increases robustly over training. (C) However, the
information speciﬁcally about the task-irrelevant subspace does compress after initially growing as
the network is trained.
information with these inputs speciﬁcally will indeed be reduced; but overall mutual information
with the input in general may still increase.
6 D ISCUSSION
Our results suggest that compression dynamics in the information plane are not a general feature of
deep networks, but are critically inﬂuenced by the nonlinearities employed by the network. Double-
saturating nonlinearities lead to compression, if mutual information is estimated by binning activations
or by adding homoscedastic noise, while single-sided saturating nonlinearities like ReLUs do not
compress in general. Consistent with this view, we ﬁnd that stochasticity in the training process does
not contribute to compression in the cases we investigate. Furthermore, we have found instances
where generalization performance does not clearly track information plane behavior, questioning the
causal link between compression and generalization. Hence information compression may parallel the
situation with sharp minima: although empirical evidence has shown a correlation with generalization
error in certain settings and architectures, further theoretical analysis has shown that sharp minima
can in fact generalize well (Dinh et al., 2017). We emphasize that compression still may occur within
a subset of the input dimensions if the task demands it. This compression, however, is interleaved
rather than in a secondary phase and may not be visible by information metrics that track the overall
information between a hidden layer and the input. Finally, we note that our results address the speciﬁc
claims of one scheme to link the information bottleneck principle with current practice in deep
networks. The information bottleneck principle itself is more general and may yet offer important
insights into deep networks (Achille & Soatto, 2017). Moreover, the information bottleneck principle
could yield fundamentally new training algorithms for networks that are inherently stochastic and
where compression is explicitly encouraged with appropriate regularization terms (Chalk et al., 2016;
Alemi et al., 2017; Kolchinsky et al., 2017).
ACKNOWLEDGMENTS
We thank Ariel Herbert-V oss for useful discussions. This work was supported by grant numbers
IIS 1409097 and CHE 1648973 from the US National Science Foundation, and by IARPA contract
#D16PC00002. Andrew Saxe and Madhu Advani thank the Swartz Program in Theoretical Theoretical
Neuroscience at Harvard University. Artemy Kolchinsky and Brendan Tracey would like to thank the
Santa Fe Institute for helping to support this research. Artemy Kolchinsky was supported by Grant
No. FQXi-RFP-1622 from the FQXi foundation and Grant No. CHE-1648973 from the US National
Science Foundation. Brendan Tracey was supported by AFOSR MURI on Multi-Information Sources
of Multi-Physics Systems under Award Number FA9550-15-1-0038.
REFERENCES
A. Achille and S. Soatto. On the Emergence of Invariance and Disentangling in Deep Representations.
arXiv preprint arXiv:1706.01350, 2017.
10

Published as a conference paper at ICLR 2018
M.S. Advani and A.M. Saxe. High-dimensional dynamics of generalization error in neural networks.
arXiv preprint arXiv:1710.03667, 2017.
A.A. Alemi, I. Fischer, J.V . Dillon, and K. Murphy. Deep variational information bottleneck. In
International Conference on Learning Representations, 2017.
P. Baldi and K. Hornik. Neural networks and principal component analysis: Learning from examples
without local minima. Neural Networks, 2:53–58, 1989.
P.L. Bartlett and S. Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural
Results. Journal of Machine Learning Research, 3:463–482, 2002.
M. Chalk, O. Marre, and G. Tkacik. Relevant sparse codes with variational information bottleneck.
In Advances in Neural Information Processing Systems, pp. 1957–1965, 2016.
G. Chechik, A. Globerson, N. Tishby, and Y . Weiss. Information bottleneck for gaussian variables.
Journal of Machine Learning Research, pp. 165–188, 2005.
J. Chee and P. Toulis. Convergence diagnostics for stochastic gradient descent with constant step size.
arXiv preprint arXiv:1710.06382, 2017.
A. Choromanska, M. Henaff, M. Mathieu, G. Arous, B., and Y . LeCun. The Loss Surfaces of
Multilayer Networks. In Proceedings of the 18th International Conference on Artiﬁcial Intelligence,
volume 38, 2015.
L. Dinh, R. Pascanu, S. Bengio, and Y . Bengio. Sharp Minima Can Generalize For Deep Nets. In
International Conference on Machine Learning, 2017.
K. Fukumizu. Effect of Batch Learning In Multilayer Neural Networks. In Proceedings of the 5th
International Conference on Neural Information Processing, pp. 67–70, 1998.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and
Y . Bengio. Generative Adversarial Nets.Advances in Neural Information Processing Systems, pp.
2672–2680, 2014.
J. Kadmon and H. Sompolinsky. Optimal Architectures in a Solvable Model of Deep Networks. In
Advances in Neural Information Processing Systems, 2016.
A. Kolchinsky and B.D. Tracey. Estimating mixture entropy with pairwise distances. Entropy, 19,
2017.
A. Kolchinsky, B.D. Tracey, and D.H. Wolpert. Nonlinear information bottleneck. arXiv preprint
arXiv:1705.02436, 2017.
A. Kraskov, H. Stögbauer, and P. Grassberger. Estimating mutual information.Physical Review E,
69:066138, 2004.
A. Krizhevsky, I. Sutskever, and G.E. Hinton. ImageNet classiﬁcation with deep convolutional neural
networks. In Advances in Neural Information Processing Systems, pp. 1097–1105. 2012.
S. Laughlin. A simple coding procedure enhances a neuron’s information capacity. Zeitschrift für
Naturforschung c, 36:910–912, 1981.
Y . LeCun, Y . Bengio, and G.E. Hinton. Deep learning.Nature, 521:436–444, 2015.
W. Lotter, G. Kreiman, and D.D. Cox. Deep predictive coding networks for video prediction and
unsupervised learning. arXiv preprint arXiv:1605.08104, 2016.
G. Montufar, R. Pascanu, K. Cho, and Y . Bengio. On the Number of Linear Regions of Deep Neural
Networks. In Advances in Neural Information Processing Systems, 2014.
N. Murata. A statistical study of on-line learning. In On-line Learning in Neural Networks, pp. 63–92.
Cambridge University Press, 1998.
11

Published as a conference paper at ICLR 2018
B. Neyshabur, R. Tomioka, and N. Srebro. Norm-Based Capacity Control in Neural Networks. In
Proceedings of The 28th Conference on Learning Theory, volume 40, pp. 1–26, 2015.
H. Poggio, T.and Mhaskar, L. Rosasco, B. Miranda, and Q. Liao. Why and when can deep-but
not shallow-networks avoid the curse of dimensionality: A review. International Journal of
Automation and Computing, pp. 1–17, 2017.
A.M. Saxe, J.L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in
deep linear neural networks. In the International Conference on Learning Representations, 2014.
J. Schmidhuber. Deep Learning in Neural Networks: An Overview. Neural Networks, 61:85–117,
2015.
H.S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples.
Physical Review A, 45:6056–6091, 1992.
R. Shwartz-Ziv and N. Tishby. Opening the black box of deep neural networks via information.arXiv
preprint arXiv:1703.00810, 2017.
D. Silver, A. Huang, C.J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser,
I. Antonoglou, V . Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner,
I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the
game of Go with deep neural networks and tree search. Nature, 529:484–489, 2016.
N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. In IEEE
Information Theory Workshop, 2015.
N. Tishby, F.C. Pereira, and W. Bialek. The information bottleneck method. Proceedings of the 37-th
Annual Allerton Conference on Communication, Control and Computing, pp. 368–377, 1999.
A L EARNING CURVES FOR tanh AND RELU NETWORKS
Supplementary Figure 7 shows the learning curves for tanh and ReLU networks depicted in Fig. 1.
Figure 7: Learning curves for (A) tanh neural network in 1 A and (B) ReLU neural network in 1 B.
Both networks show good generalization with regards to the test data.
B R OBUSTNESS OF FINDINGS TO MI ESTIMATION METHOD AND NEURAL
ACTIVATION FUNCTIONS
This Appendix investigates the generality of the ﬁnding that compression is not observed in neural
network layers with certain activation functions. Figure 1 of the main text shows example results
using a binning-based MI estimator and a nonparametric KDE estimator, for both thetanh and ReLU
activation functions. Here we describe the KDE MI estimator in detail, and present extended results
on other datasets. We also show results for other activation functions. Finally, we provide entropy
estimates based on another nonparametric estimator, the popular k-nearest neighbor approach of
Kraskov et al. (2004). Our ﬁndings consistently show that double-saturating nonlinearities can yield
compression, while single-sided nonlinearities do not.
12