On the Information Bottleneck Theory of Deep Learning (Saxe et al.) — part 5/6

Cleaned-text code points [57067, 71867). Bibliography omitted here; source.md includes it.
Published as a conference paper at ICLR 2018
Gamma function, andψ(·) is the digamma function. Here the parameter ϵ prevents inﬁnite terms
when the nearest neighbor distanceri = 0 for some sample. We tookϵ = 10−16.
Figure 12 shows the entropy over training for tanh and ReLU networks trained on the dataset of
and with the network architecture in Shwartz-Ziv & Tishby (2017), averaged over 50 repeats. In
these experiments, we usedk = 2. Compression would correspond to decreasing entropy over the
course of training, while a lack of compression would correspond to increasing entropy. Severaltanh
layers exhibit compression, while the ReLU layers do not. Hence qualitatively, the Kraskov estimator
returns similar results to the binning and KDE strategies.
A B
Figure 12: Entropy dynamics over training for the network architecture and training dataset of
Shwartz-Ziv & Tishby (2017), estimated with the nonparametric k-nearest-neighbor-based method of
Kraskov et al. (2004). Here the x-axis is epochs of training time, and the y-axis plots the entropy of
the hidden representation, as calculated using nearest-neighbor distances. Note that in this setting, if
T is considered to be the hidden activity plus independent noise, the entropy is equal to the mutual
information up to a constant (see derivation in text). Layers 0-4 correspond to the hidden layers of
size 10-7-5-4-3. (A) tanh neural network layers can show compression over the course of training.
(B) ReLU neural network layers show no compression.
C N OISE ASSUMPTIONS AND DISCRETE VS CONTINUOUS ENTROPY
A recurring theme in the results reported in this paper is the necessity of noise assumptions to yield a
nontrivial information theoretic analysis. Here we give an extended discussion of this phenomenon,
and of issues relating to discrete entropy as opposed to continuous (differential) entropy.
The activity of a neural network is often a continuous deterministic function of its input. That is, in
response to an inputX, a speciﬁc hidden layer might produce activityh =f(X) for some function
f. The mutual information betweenh andX is given by
I(h;X) = H(h)−H(h|X). (17)
Ifh were a discrete variable, then the entropy would be given by
H(h) =−
N∑
i=1
pi logpi (18)
wherepi is the probability of the discrete symboli, as mentioned in the main text. ThenH(h|X) = 0
because the mapping is deterministic and we haveI(h;X) =H(h).
Howeverh is typically continuous. The continuous entropy, deﬁned for a continuous random variable
Z with densitypZ by analogy to Eqn. (18) as
H(Z) =−
∫
pZ(z) logpZ(z)dz, (19)
can be negative and possibly inﬁnite. In particular, note that ifpZ is a delta function, thenH(Z) =
−∞. The mutual information between hidden layer activityh and the inputX for continuoush,X is
I(h;X) =H(h)−H(h|X). (20)
19

Published as a conference paper at ICLR 2018
NowH(h|X) =−∞ since given the inputX, the hidden activityh is distributed as a delta function
atf(X). The mutual information is thus generally inﬁnite, so long as the hidden layer activity has
ﬁnite entropy (H(h) is ﬁnite).
Figure 13: Effect of binning strategy on minimal three neuron model. Mutual information for the
simple three neuron model shown in Fig. 2 with bin edges bi∈ tanh(linspace(−50, 50,N )). In
contrast to linear binning, the mutual information continues to increase as weights grow.
To yield a ﬁnite mutual information, some noise in the mapping is required such thatH(h|X) remains
ﬁnite. A common choice (and one adopted here for the linear network, the nonparametric kernel
density estimator, and the k-nearest neighbor estimator) is to analyze a new variable with additive
noise,T =h +Z, whereZ is a random variable independent ofX. ThenH(T|X) =H(Z) which
allows the overall informationI(T ;X) =H(T )−H(Z) to remain ﬁnite. This noise assumption is
not present in the actual neural networks either during training or testing, and is made solely for the
purpose of calculating the mutual information.
Another strategy is to partition the continuous variable h into a discrete variable T, for instance
by binning the values (the approach taken in Shwartz-Ziv & Tishby (2017)). This allows use
of the discrete entropy, which remains ﬁnite. Again, however, in practice the network does not
operate on the binned variablesT but on the continuous variablesh, and the binning is solely for
the purpose of calculating the mutual information. Moreover, there are many possible binning
strategies, which yield different discrete random variables, and different mutual information with
respect to the input. The choice of binning strategy is an assumption analogous to choosing a type
of noise to add to the representation in the continuous case: because there is in fact no binning in
the operation of the network, there is no clear choice for binning methodology. The strategy we use
in binning-based experiments reported here is the following: for bounded activations like the tanh
activation, we use evenly spaced bins between the minimum and maximum limits of the function. For
unbounded activations like ReLU, we ﬁrst train the network completely; next identify the minimum
and maximum hidden activation over all units and all training epochs; and ﬁnally bin into equally
spaced bins between these minimum and maximum values. We note that this procedure places no
restriction on the magnitude that the unbounded activation function can take during training, and
yields the same MI estimate as using inﬁnite equally spaced bins (because bins for activities larger
than the maximum are never seen during training).
As an example of another binning strategy that can yield markedly different results, we consider
evenly spaced bins in a neuron’s net input, rather than its activity. That is, instead of evenly spaced
bins in the neural activity, we determine the bin edges by mapping a set of evenly spaced values
through the neural nonlinearity. For tanh, for instance, this spaces bins more tightly in the saturation
region as compared to the linear region. Figure 13 shows the results of applying this binning strategy
20

Published as a conference paper at ICLR 2018
Figure 14: Effect of binning strategy on information plane dynamics. Results for the same tanh
network and training regime as 1A, but with bin edgesbi∈ tanh(linspace(−50, 50,N )). Measured
with this binning structure, there is no compression in most layers.
to the minimal three neuron model with tanh activations. This binning scheme captures more
information as the weights of the network grow larger. Figure 14 shows information plane dynamics
for this binning structure. The tanh network no longer exhibits compression. (We note that the
broken DPI in this example is an artifact of performing binning only for analysis, as discussed below).
Any implementation of a neural network on digital hardware is ultimately of ﬁnite precision, and
hence is a binned, discrete representation. However, it is a very high resolution binning compared
to that used here or by Shwartz-Ziv & Tishby (2017): single precision would correspond to using
roughly 232 bins to discretize each hidden unit’s activity, as compared to the 30-100 used here. If the
binning is ﬁne-grained enough that each input X yields a different binned activity patternh, then
H(h) = log(P ) whereP is the number of examples in the dataset, and there will be little to no
change in information during training. As an example, we show in Fig. 15 the result of binning at full
machine precision.
Finally, we note two consequences of the assumption of noise/binning for the purposes of analysis.
First, this means that the data processing inequality (DPI) does not apply to the noisy/binned mutual
information estimates. The DPI states that information can only be destroyed through successive
transformations, that is, ifX→h1→h2 form a Markov chain, thenI(X;h1)≥I(X;h2) (see, eg,
Tishby & Zaslavsky (2015)). Because noise is added only for the purpose of analysis, however, this
does not apply here. In particular, for the DPI to apply, the noise added at lower layers would have to
propagate through the network to higher layers. That is, if the transformation from hidden layer 1
to hidden layer 2 is h2 =f(h1) andT1 =h1 +Z1 is the hidden layer activity after adding noise,
then the DPI would hold for the variable ˜T2 = f(T1) +Z2 = f(h1 +Z1) +Z2, not the quantity
21

Published as a conference paper at ICLR 2018
I(
A B
I(Y;T)I(X;T)
I(X;T)
I(Y;T)
Figure 15: Effect of binning at full machine precision. (A) ReLU network. (B) tanh network.
Information in most layers stays pinned to log2(P ) = 12 . Compression is only observed in the
highest and smallest layers near the very end of training, when the saturation oftanh is strong enough
to saturate machine precision.
T2 = h2 +Z2 = f(h1) +Z2 used in the analysis. Said another way, the Markov chain for T2 is
X→h1→h2→T2, notX→h1→T1→T2, so the DPI states only thatI(X;h1)≥I(X;T2).
A second consequence of the noise assumption is the fact that the mutual information is no longer
invariant to invertible transformations of the hidden activity h. A potentially attractive feature
of a theory based on mutual information is that it can allow for comparisons between different
architectures: mutual information is invariant to any invertible transformation of the variables, so two
hidden representations could be very different in detail but yield identical mutual information with
respect to the input. However, once noise is added to a hidden representation, this is no longer the case:
the variableT =h +Z is not invariant to reparametrizations ofh. As a simple example, consider
a minimal linear network with scalar weights w1 andw2 that computes the output ˆy = w2w1X.
The hidden activity ish =w1X. Now consider the family of networks in which we scale downw1
and scale upw2 by a factorc⁄= 0, that is, these networks have weights ˜w1 =w1/c and ˜w2 =cw2,
yielding the exact same input-output map ˆy = ˜w2 ˜w1X =cw2(w1/c)X =w2w1X. Because they
compute the same function, they necessarily generalize identically. However after introducing the
noise assumption the mutual information is
I(T ;X) = log
(
w2
1/c2 +σ2
M I
)
− log
(
σ2
M I
)
(21)
where we have taken the setting in Section 3 in which X is normal Gaussian, and independent
Gaussian noise of varianceσ2
M I is added for the purpose of MI computation. Clearly, the mutual
information is now dependent on the scalingc of the internal layer, even though this is an invertible
linear transformation of the representation. Moreover, this shows that networks which generalize
identically can nevertheless have very different mutual information with respect to the input when it
is measured in this way.
D W EIGHT NORMS OVER TRAINING
Our argument relating neural saturation to compression in mutual information relies on the notion
that in typical training regimes, weights begin small and increase in size over the course of training.
We note that this is a virtual necessity for a nonlinearity like tanh, which is linear around the origin:
when initialized with small weights, the activity of a tanh network will be in this linear regime and
the network can only compute a linear function of its input. Hence a real world nonlinear task can
only be learned by increasing the norm of the weights so as to engage the tanh nonlinearity on some
examples. This point can also be appreciated from norm-based capacity bounds on neural networks,
which show that, for instance, the Rademacher complexity of a neural network with small weights
must be low (Bartlett & Mendelson, 2002; Neyshabur et al., 2015). Finally, as an empirical matter,
the networks trained in this paper do in fact increase the norm of their weights over the course of
22

Published as a conference paper at ICLR 2018
training, as shown by the green lines in Figure 20 for tanh and ReLU networks in the training setting
of Shwartz-Ziv & Tishby (2017); Figures 9 and 10 for the MNIST networks; and Figure 21 for a
linear network.
E H ISTOGRAMS OF NEURAL ACTIVATIONS
Supplementary Figures 16 and 17 show histograms of neural activities over the course of training in
tanh and ReLU networks respectively.
Figure 16: Histogram of neural activities in a tanh network during training. The ﬁnal three layers
eventually saturate in the top and bottom bins corresponding to the saturation limits of the tanh
activation function, explaining the compression observed in tanh. x-axis: training time in epochs.
y-axis: Hidden activity bin values from lowest to highest. Colormap: density of hidden layer activities
across all input examples.
23

Published as a conference paper at ICLR 2018
Figure 17: Histogram of neural activities in a ReLU network during training. ReLU layers 1-5 have
a roughly constant fraction of activities at zero, corresponding to instances where the ReLU is off;
the nonzero activities disperse over the course of training without bound, yielding higher entropy
distributions. The sigmoid output layer 6 converges to its saturation limits, and is the only layer that
compresses during training (c.f. Fig. 1B). x-axis: training time in epochs. y-axis: Hidden activity
value. Colormap: density of hidden layer activities across all input examples.
F I NFORMATION PLANE DYNAMICS IN DEEPER LINEAR NETWORKS
Supplementary Figure 18 shows information plane dynamics for a deep neural network with ﬁve
hidden layers each containing 50 hidden units.
24

Published as a conference paper at ICLR 2018
Train
Test
A B
Figure 18: Information plane dynamics in a deep linear neural network. (A) Train and test error
during learning. (B) Information plane dynamics. No compression is visible.
G L INEAR MUTUAL INFORMATION CALCULATION
For the linear setting considered here, the mutual information between a hidden representationT and
the outputY may be calculated using the relations
H(Y ) = No
2 log(2πe) + 1
2 log|WoW T
o +σ2
oINo|, (22)
H(T ) = Nh
2 log(2πe) + 1
2 log| ¯W ¯W T +σ2
M IINh|, (23)
H(Y ;T ) = No +Nh
2 log(2πe) + 1
2 log
⏐⏐⏐⏐
¯W ¯W T +σ2
M IINh
¯WW T
o ,
Wo ¯W T WoW T
o +σ2
oINh
⏐⏐⏐⏐, (24)
I(Y ;T ) =H(Y ) +H(T )−H(Y ;T ). (25)
H S TOCHASTIC VS BATCH TRAINING
Figure 19 shows information plane dynamics for stochastic and batch gradient descent learning in a
linear network. Randomness in the training process does not dramatically alter the information plane
dynamics.
Figure 19: Effect of stochastic training in linear networks. (A) Information plane dynamics for
stochastic gradient descent in a linear network (same setting as Fig. 4). (B) Information plane
dynamics for batch gradient descent.
I G RADIENT SNR PHASE TRANSITION
The proposed mechanism of compression in Shwartz-Ziv & Tishby (2017) is noise arising from
stochastic gradient descent training. The results in Section 4 of the main text show that compression
25