On the Information Bottleneck Theory of Deep Learning (Saxe et al.) — part 6/6

Cleaned-text code points [71867, 76602). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2018
still occurs under batch gradient descent learning, suggesting that in fact noise in the gradient updates
is not the cause of compression. Here we investigate a related claim, namely that during training,
networks switch between two phases. These phases are defined by the ratio of the mean of the gradient
to the standard deviation of the gradient across training examples, called the gradient signal-to-noise
ratio. In the first “drift” phase, the SNR is high, while in the second “diffusion” phase the SNR
is low. Shwartz-Ziv & Tishby (2017) hypothesize that the drift phase corresponds to movement
toward the minimum with no compression, while the diffusion phase corresponds to a constrained
diffusion in weight configurations that attain the optimal loss, during which representations compress.
However, two phases of gradient descent have been described more generally, sometimes known as
the transient and stochastic phases or search and convergence phases (Murata, 1998; Chee & Toulis,
2017), suggesting that these phases might not be related specifically to compression behavior.
In Fig. 20 we plot the gradient SNR over the course of training for the tanh and ReLU networks in
the standard setting of Shwartz-Ziv & Tishby (2017). In particular, for each layerl we calculate the
mean and standard deviation as
ml =
‖‖‖‖
⟨ ∂E
∂Wl
⟩‖‖‖‖
F
(26)
sl =
‖‖‖‖STD
( ∂E
∂Wl
)‖‖‖‖
F
(27)
where⟨·⟩ denotes the mean and STD (·) denotes the element-wise standard deviation across all
training samples, and‖·‖F denotes the Frobenius norm. The gradient SNR is then the ratioml/sl.
We additionally plot the norm of the weights‖Wl‖F over the course of training.
Both tanh and ReLU networks yield a similar qualitative pattern, with SNR undergoing a step-like
transition to a lower value during training. Figures 9 and 10, fourth row, show similar plots for
MNIST-trained networks. Again, SNR undergoes a transition from high to low over training. Hence
the two phase nature of gradient descent appears to hold across the settings that we examine here.
Crucially, this finding shows that the SNR transition is not related to the compression phenomenon
because ReLU networks, which show the gradient SNR phase transition, do not compress.
Finally, to show the generality of the two-phase gradient SNR behavior and its independence from
compression, we develop a minimal model of this phenomenon in a three neuron linear network. We
consider the student-teacher setting of Fig. 3 but withNi =Nh = 1, such that the input and hidden
layers have just a single neuron (as in the setting of Fig. 2). Here, with just a single hidden neuron,
clearly there can be no compression so long as the first layer weight increases over the course of
training. Figure 21AC shows that even in this simple setting, the SNR shows the phase transition but
the weight norm increases over training. Hence again, the two phases of the gradient are present even
though there is no compression. To intuitively understand the source of this behavior, note that the
weights are initialized to be small and hence early in learning all must be increased in magnitude,
yielding a consistent mean gradient. Once the network reaches the vicinity of the minimum, the mean
weight change across all samples by definition goes to zero. The standard deviation remains finite,
however, because on some specific examples error could be improved by increasing or decreasing the
weights–even though across the whole dataset the mean error has been minimized.
Hence overall, our results show that a two-phase structure in the gradient SNR occurs in all settings we
consider, even though compression occurs only in a subset. The gradient SNR behavior is therefore
not causally related to compression dynamics, consistent with the view that saturating nonlinearities
are the primary source of compression.
26

Published as a conference paper at ICLR 2018
Figure 20: Gradient SNR phase transition. (A) tanh networks trained in the standard setting of
Shwartz-Ziv & Tishby (2017) show a phase transition in every layer. (B) ReLU networks also show a
phase transition in every layer, despite exhibiting no compression.
Error
Error
B
C
A
D
Figure 21: Minimal model exhibiting gradient SNR phase transition. Here a three neuron linear
network (architecture 1− 1− 1) learns to approximate a teacher. Other parameters are teacher
SNR = 1, number of training samplesP = 100, learning rate.001. Left column: (A) The loss over
training with SGD (minibatch size 1). (C) The resulting gradient SNR dynamics. Right column:
(B) The loss over training with BGD. (D) The resulting gradient SNR dynamics averaging over all
training samples (not minibatches, see text).
27