On the Information Bottleneck Theory of Deep Learning (Saxe et al.) — part 4/6

Cleaned-text code points [43699, 57067). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2018
B.1 K ERNEL DENSITY ESTIMATION OF MI
The KDE approach of Kolchinsky & Tracey (2017); Kolchinsky et al. (2017) estimates the mutual
information between the input and the hidden layer activity by assuming that the hidden activity is
distributed as a mixture of Gaussians. This assumption is well-suited to the present setting under
the following interpretation: we take the input activity to be distributed as delta functions at each
example in the dataset, corresponding to a uniform distribution over these specific samples. In other
words, we assume that the empirical distribution of input samples is the true distribution. Next, the
hidden layer activityh is a deterministic function of the input. As mentioned in the main text and
discussed in more detail in Appendix C, without the assumption of noise, this would have infinite
mutual information with the input. We therefore assume for the purposes of analysis that Gaussian
noise of varianceσ2 is added, that is,T =h +ϵ whereϵ∼N (0,σ 2I). Under these assumptions,
the distribution ofT is genuinely a mixture of Gaussians, with a Gaussian centered on the hidden
activity corresponding to each input sample. We emphasize again that the noiseϵ is added solely for
the purposes of analysis, and is not present during training or testing the network. In this setting, an
upper bound for the mutual information with the input is (Kolchinsky & Tracey, 2017; Kolchinsky
et al., 2017)
I(T ;X)≤− 1
P
∑
i
log 1
P
∑
j
exp
(
−1
2
||hi−hj||2
2
σ2
)
(7)
whereP is the number of training samples andhi denotes the hidden activity vector in response to
input samplei. Similarly, the mutual information with respect to the output can be calculated as
I(T ;Y ) = H(T )−H(T|Y ) (8)
≤ − 1
P
∑
i
log 1
P
∑
j
exp
(
−1
2
‖hi−hj‖2
2
σ2
)
(9)
−
L∑
l
pl

− 1
Pl
∑
i,Yi=l
log 1
Pl
∑
j,Yj =l
exp
(
−1
2
‖hi−hj‖2
2
σ2
)
 (10)
whereL is the number of output labels,Pl denotes the number of data samples with output labell,
pl =Pl/P denotes the probability of output labell, and the sums overi,Y i =l indicate a sum over
all examples with output labell.
13

Published as a conference paper at ICLR 2018
Figure 8: Information plane dynamics for the network architecture and training dataset of Shwartz-
Ziv & Tishby (2017), estimated with the nonparametric KDE method of Kolchinsky & Tracey
(2017); Kolchinsky et al. (2017) and averaged over 50 repetitions. (A) tanh neural network layers
show compression. (B) ReLU neural network layers show no compression. (C) The soft-sign
activation function, a double-saturating nonlinearity that saturates more gently than tanh, shows
modest compression. (D) The soft-plus activation function, a smoothed version of the ReLU, exhibits
no compression. Hence double-saturating nonlinearities exhibit the compression effect while single-
saturating nonlinearities do not.
Figure 8A-B shows the result of applying this MI estimation method on the dataset and network
architecture of Shwartz-Ziv & Tishby (2017), with MI estimated on the full dataset and averaged
over 50 repetitions. Mutual information was estimated using data samples from the test set, and we
took the noise varianceσ2 = 0.1. These results look similar to the estimate derived from binning,
with compression in tanh networks but no compression in ReLU networks. Relative to the binning
estimate, it appears that compression is less pronounced in the KDE method.
Figure 1C-D of the main text shows the results of this estimation technique applied to a neural network
of size 784− 1024− 20− 20− 20− 10 on the MNIST handwritten digit classification dataset.
The network was trained using SGD with minibatches of size 128. As before, mutual information
was estimated using data samples from the test set, and we took the noise varianceσ2 = 0.1. The
smaller layer sizes in the top three hidden layers were selected to ensure the quality of the kernel
density estimator given the amount of data in the test set, since the estimates are more accurate for
smaller-dimensional data. Because of computational expense, the MNIST results are from a single
training run.
More detailed results for the MNIST dataset are provided in Figure 9 for the tanh activation function,
and in Figure 10 for the ReLU activation function. In these figures, the first row shows the evolution
of the cross entropy loss (on both training and testing data sets) during training. The second row
shows the mutual information between input and the activity of different hidden layers, using the
nonparametric KDE estimator described above. The blue region in the second row shows the range
of possible MI values, ranging from the upper bound described above (Eq. 10) to the following lower
14

Published as a conference paper at ICLR 2018
bound (Kolchinsky & Tracey, 2017),
I(T ;Y ) ≥ − 1
P
∑
i
log 1
P
∑
j
exp
(
−1
2
‖hi−hj‖2
2
4σ2
)
(11)
−
L∑
l
pl

− 1
Pl
∑
i,Yi=l
log 1
Pl
∑
j,Yj =l
exp
(
−1
2
‖hi−hj‖2
2
4σ2
)
. (12)
The third row shows the mutual information between input and activity of different hidden layers,
estimated using the binning method (here, the activity of each neuron was discretized into bins of
size 0.5). For both the second and third rows, we also plot the entropy of the inputs, H(X), as a
dashed line. H(X) is an upper bound on the mutual information I(X;T ), and is computed using
the assumption of a uniform distribution over the 10,000 testing points in the MNIST dataset, giving
H(X) = log2 10000.
Finally, the fourth row visualizes the dynamics of the SGD updates during training. For each layer and
epoch, the green line shows the𝓁2 norm of the weights. We also compute the vector of mean updates
across SGD minibatches (this vector has one dimension for each weight parameter), as well as the
vector of the standard deviation of the updates across SGD minibatches. The𝓁2 norm of the mean
update vector is shown in blue, and the𝓁2 norm of the standard deviation vector is shown in orange.
The gradient SNR, computed as the ratio of the norm of the mean vector to the norm of the standard
deviation vector, is shown in red. For both the tanh and ReLU networks, the gradient SNR shows a
phase transition during training, and the norm of the weights in each layer increases. Importantly,
this phase transition occurs despite a lack of compression in the ReLU network, indicating that noise
in SGD updates does not yield compression in this setting.
15

Published as a conference paper at ICLR 2018
100 101 102 103 104
Epoch
0.0
0.5
1.0
1.5
2.0
2.5Cross/uni00A0entropy/uni00A0loss
Loss
Train
Test
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A01/uni00A0Mutual/uni00A0Info/uni00A0(KDE)
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A01/uni00A0Mutual/uni00A0Info/uni00A0(binned)
100 101 102 103 104
Epoch
10/uni00AD4
10/uni00AD3
10/uni00AD2
10/uni00AD1
100
101
102 Layer/uni00A01/uni00A0SNR
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A02/uni00A0Mutual/uni00A0Info/uni00A0(KDE)
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A02/uni00A0Mutual/uni00A0Info/uni00A0(binned)
100 101 102 103 104
Epoch
10/uni00AD5
10/uni00AD4
10/uni00AD3
10/uni00AD2
10/uni00AD1
100
101 Layer/uni00A02/uni00A0SNR
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A03/uni00A0Mutual/uni00A0Info/uni00A0(KDE)
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A03/uni00A0Mutual/uni00A0Info/uni00A0(binned)
100 101 102 103 104
Epoch
10/uni00AD5
10/uni00AD4
10/uni00AD3
10/uni00AD2
10/uni00AD1
100
101 Layer/uni00A03/uni00A0SNR
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A04/uni00A0Mutual/uni00A0Info/uni00A0(KDE)
H(X)
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A04/uni00A0Mutual/uni00A0Info/uni00A0(binned)
H(X)
100 101 102 103 104
Epoch
10/uni00AD5
10/uni00AD4
10/uni00AD3
10/uni00AD2
10/uni00AD1
100
101 Layer/uni00A04/uni00A0SNR
Mean
Std
SNR
||W||
Figure 9: Detailed tanh activation function results on MNIST. Row 1: Loss over training. Row
2: Upper and lower bounds for the mutual information I(X;T ) between the input (X) and each
layer’s activity (T ), using the nonparametric KDE estimator (Kolchinsky & Tracey, 2017; Kolchinsky
et al., 2017). Dotted line indicatesH(X) = log2 10000, the entropy of a uniform distribution over
10,000 testing samples. Row 3: Binning-based estimate of the mutual information I(X;T ), with
each neuron’s activity discretized using a bin size of 0.5. Row 4: Gradient SNR and weight norm
dynamics. The gradient SNR shows a phase transition during training, and the norm of the weights in
each layer increases.
16

Published as a conference paper at ICLR 2018
100 101 102 103 104
Epoch
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5Cross/uni00A0entropy/uni00A0loss
Loss
Train
Test
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A01/uni00A0Mutual/uni00A0Info/uni00A0(KDE)
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A01/uni00A0Mutual/uni00A0Info/uni00A0(binned)
100 101 102 103 104
Epoch
10/uni00AD5
10/uni00AD4
10/uni00AD3
10/uni00AD2
10/uni00AD1
100
101
102 Layer/uni00A01/uni00A0SNR
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A02/uni00A0Mutual/uni00A0Info/uni00A0(KDE)
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A02/uni00A0Mutual/uni00A0Info/uni00A0(binned)
100 101 102 103 104
Epoch
10/uni00AD5
10/uni00AD4
10/uni00AD3
10/uni00AD2
10/uni00AD1
100
101 Layer/uni00A02/uni00A0SNR
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A03/uni00A0Mutual/uni00A0Info/uni00A0(KDE)
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A03/uni00A0Mutual/uni00A0Info/uni00A0(binned)
100 101 102 103 104
Epoch
10/uni00AD5
10/uni00AD4
10/uni00AD3
10/uni00AD2
10/uni00AD1
100
101 Layer/uni00A03/uni00A0SNR
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A04/uni00A0Mutual/uni00A0Info/uni00A0(KDE)
H(X)
100 101 102 103 104
Epoch
0
2
4
6
8
10
12
14
I(X; T)
Layer/uni00A04/uni00A0Mutual/uni00A0Info/uni00A0(binned)
H(X)
100 101 102 103 104
Epoch
10/uni00AD5
10/uni00AD4
10/uni00AD3
10/uni00AD2
10/uni00AD1
100
101 Layer/uni00A04/uni00A0SNR
Mean
Std
SNR
||W||
Figure 10: Detailed ReLU activation function results on MNIST. Row 1: Loss over training. Row
2: Upper and lower bounds for the mutual information I(X;T ) between the input (X) and each
layer’s activity (T ), using the nonparametric KDE estimator (Kolchinsky & Tracey, 2017; Kolchinsky
et al., 2017). Dotted line indicatesH(X) = log2 10000, the entropy of a uniform distribution over
10,000 testing samples. Row 3: Binning-based estimate of the mutual information I(X;T ), with
each neuron’s activity discretized using a bin size of 0.5. Row 4: Gradient SNR and weight norm
dynamics. The gradient SNR shows a phase transition during training, and the norm of the weights in
each layer increases. Importantly, this phase transition occurs despite a lack of compression in the
ReLU network, indicating that noise in SGD updates does not yield compression in this setting.
17

Published as a conference paper at ICLR 2018
-5 0 5
Net input
-1
0
1
2
3
4
5
6
Activation
tanh
ReLU
Softsign
Softplus
Figure 11: Alternative activation functions.
B.2 O THER ACTIVATION FUNCTIONS
Next, in Fig. 8C-D, we show results from the kernel MI estimator from two additional nonlinear
activation functions, the softsign function
f(x) = x
1 +|x|,
and the softplus function
f(x) = ln(1 +ex).
These functions are plotted next to tanh and ReLU in Fig. 11. The softsign function is similar to
tanh but saturates more slowly, and yields less compression than tanh. The softplus function is a
smoothed version of the ReLU, and yields similar dynamics with no compression. Because softplus
never saturates fully to zero, it retains more information with respect to the input than ReLUs in
general.
B.3 K RASKOV ESTIMATOR
We additionally investigated the widely-used nonparametric MI estimator of Kraskov et al. (2004).
This estimator uses nearest neighbor distances between samples to compute an estimate of the
entropy of a continuous random variable. Here we focused for simplicity only on the compression
phenomenon in the mutual information between the input and hidden layer activity, leaving aside the
information with respect to the output (as this is not relevant to the compression phenomenon). Again,
without additional noise assumptions, the MI between the hidden representation and the input would
be infinite because the mapping is deterministic. Rather than make specific noise assumptions, we
instead use the Kraskov method to estimate the entropy of the hidden representationsT . Note that the
entropy ofT is the mutual information up to an unknown constant so long as the noise assumption is
homoscedastic, that is,T =h +Z where the random variableZ is independent ofX. To see this,
note that
I(T ;X) = H(T )−H(T|X) (13)
= H(T )−H(Z) (14)
= H(T )−c (15)
where the constantc =H(Z). Hence observing compression in the layer entropy H(T ) is enough to
establish that compression occurs in the mutual information.
The Kraskov estimate is given by
d
P
P∑
i=1
log(ri +ϵ) +d
2 log(π)− log Γ(d/2 + 1) +ψ(P )−ψ(k) (16)
whered is the dimension of the hidden representation,P is the number of samples,ri is the distance
to the k-th nearest neighbor of sample i,ϵ is a small constant for numerical stability, Γ(·) is the
18