Comparing representational geometries using whitened unbiased-distance-matrix similarity — part 3/6

Cleaned-text code points [30887, 46745). Bibliography omitted here; source.md includes it.

To determine the influence of RDM whitening in the context of realistic model comparisons, we simulated data using the designs of three published fMRI experiments and evaluated the associated models, all of which made quantitative predictions about representational distances (Fig. 6). The first two experiments measured activation patterns associated with 5 (Exp 1) or 31 (Exp 2) different finger movements in primary motor cortex [ 9 ] . The associated models where derived either from the structure of the associated muscle activity or from the natural statistics of movement. The third experiment measured the activation patterns elicited by 92 images showing a range of animate and inanimate objects in human inferior temporal cortex. The models were derived from the 8 layers of a neural network [ 20 ] .

As for Figure 4, we simulated data from each model, using different signal-to-noise levels. We then used RDM correlations or whitened RDM correlations, as well as cosine similarity or whitened cosine similarity to find the best model. For each method we recorded the number of correct model decisions. In addition to the RSA-based methods, we also used Pattern Component Modeling (PCM) [ 14 , 19 ] , which directly compares the marginal likelihood of the data given the models, under the assumption that both signal and noise are normally distributed. For such data, PCM implements the likelihood-ratio test between models. For the case of our simulations, where all assumptions hold, PCM therefore implements the optimal inference procedure [ 13 ] and provides an upper performance bound for any model-comparison technique.

Across the three different experimental simulations, the simple RDM correlation or RDM cosine similarity (Fig. 6, solid lines) performed clearly sub-optimally as compared to PCM. Taking the covariance structure of the distances into account (dashed lines) substantially improved model decisions. Indeed, in many cases, the performance of RSA inference was close to optimal. This suggests that accounting for the signal-dependent, second half of the covariance formula (Eq. 27 , 29 ) would not improve inference much further. Instead, these simulations indicate that the observed sub-optimal performance was mostly caused by the assumption that the distances are uncorrelated, rather than by the assumption that the distances have the same variance.

### 2.10 Factors influencing the advantage of RDM whitening

From the simulations in Figure 6, it appears that the importance of taking the covariance structure of the distances estimates into account is more pronounced for experiments with more conditions. To test this idea directly, we simulated data for Exp 1 with different numbers of conditions. We started with 5 conditions and 32 partitions. We then reanalyzed the data, relabeling the measures from even partitions as conditions 1-5, and from the odd partitions as condition 6-10. This increases the number of conditions to 10 and reduces the number of partitions to 16. We repeated this procedure two times more, finally ending up with 40 conditions and 4 partitions. As the underlying data is the same, the performance of PCM was relatively constant across these situations. With increasing number of conditions, however, the advantage of using WUC over the normal RDM cosine similarity increases (Fig. 7). This may appear at first somewhat counter-intuitive, as the proportion of uncorrelated distances pairs in the RDM increases with increasing number of conditions. However, the structure of the covariance matrix (Fig. 5b) is such that the anisotropy increases with the number of conditions. The axis of highest variability of the distance estimates is always in the direction of the average of all distances. This direction is associated with an eigenvalue of K K . There are also K − 1 K-1 orthogonal directions with an eigenvalue of K / 2 K/2 , and K ⁡ ( K − 3 ) K(K-3) directions with an eigenvalue of 1. Thus, the ratio of the larger to the smaller eigenvalues of 𝐕 \mathbf{V} (a measure of anisotropy) scales linearly with the number of conditions. That means that with 40 conditions, the all-mean dimension has 40 times higher variability than most of the other directions. When ignoring the covariance structure, all dimensions are counted to be equally important, which leads to sub-optimal inferences.

This consideration also implies that for some model comparison problems, taking into account the covariance will not change the inference. This is the case when two models differ only on dimensions of model space that can be measured with equal variability (i.e., have the same eigenvalue in 𝐕 \mathbf{V} ). The two models compared in Fig. 4a are such an example. Most model comparison problems, however, will improve, and inference will never get worse. Thus, using whitened RDM similarities is always recommended.

### 2.11 Interpretation of the WUC

An important consideration in the interpretation of the WUC is that its expected value will be zero if and only if all distances that are systematically larger than zero in the data are predicted to have a distance of zero in the model. That is, a WUC of zero shows that there is no above-chance linear prediction possible from the features of the model to the activity patterns (see Fig. 1).

Therefore, any model that predicts a positive distance for any pair of conditions that are distinct in the data, will have a WUC > 0 >0 . In contrast, the traditional RDM Pearson correlation will be zero when there is no systematic relationship between measured and predicted dissimilarities. As a consequence, the values of the WUC tend to be substantially higher than RDM correlations, often close to 1 for most models.

To obtain a baseline that would be equivalent to a correlation of zero, it is therefore often useful to compare all models to a null-model that predicts all conditions as equally distinct from each other. This approach combines the interpretability of the RDM correlation with the increased power of the WUC.

### 2.12 Extension of the WUC to flexible RDM models

In many applications, we want to fit and evaluate flexible models, in which the vector of predicted dissimilarities depends on a vector of model parameters 𝜽 \bm{\theta} . In some cases the predicted RDM is a weighted sum of different model components, for example the different layers of a deep neural network [ 20 , 21 ] . In other cases, the predicted distances are a non-linear function of the model parameters [ 19 ] , for example when parameterizing the width of a tuning function in a population-receptive-field model [ 22 ] . In all of these cases, we can take into account the covariance of the dissimilarity estimates by minimizing the following loss function:

J ⁡ ( 𝜽 ) = ( 𝐝 − 𝐦 ⁡ ( 𝜽 ) ) T ​ 𝐕 − 1 ​ ( 𝐝 − 𝐦 ⁡ ( 𝜽 ) ) . J(\bm{\theta})=(\mathbf{d}-\mathbf{m(\bm{\theta})})^{T}\mathbf{V}^{-1}(\mathbf{d}-\mathbf{m(\bm{\theta})}). (13)

Equivalently, we can whiten the estimation error by pre-multiplying both estimated distances and the model prediction with 𝐕 − 1 2 \mathbf{V}^{-\frac{1}{2}} , and then use standard least squares approaches.

### 2.13 Relationship to other multivariate dependence measures

Interestingly, the whitened RDM cosine similarity is related to other statistical measures of multivariate dependence. If a whitened RDM cosine similarity is calculated from biased distance estimates, and if we assume i.i.d. measurement noise, it is identical linear Centered Kernel Alignment (CKA, [ 12 , 23 ] ), also known as the RV coefficient [ 11 ] . The CKA is a normalized version of the Hilbert-Schmidt independence criterion (HSIC [ 24 ] ) between two sets of multivariate patterns. In general, the HSIC is the dot-product between the elements of two similarity (or kernel) matrices. Here we have the special linear case, where the similarity is the dot product of the two pattern vectors.

Let 𝐀 \mathbf{A} and 𝐁 \mathbf{B} be two matrices, with the same number K K of rows, containing patterns for K K observations (e.g., trials or time points). To make the mean of each column of these matrices equal to zero, we can pre-multiply the patterns with the centering matrix 𝐇 = 𝐈 K − 𝟏 K / K \mathbf{H}=\mathbf{I}_{K}-\mathbf{1}_{K}/K . We then can define the centred second moment matrix of the patterns as

𝐆 𝐀 = 𝐇𝐀𝐀 T ​ 𝐇 T / P 𝐆 𝐁 = 𝐇𝐁𝐁 T ​ 𝐇 T / P . \mathbf{G_{A}}=\mathbf{H}\mathbf{A}\mathbf{A}^{T}\mathbf{H}^{T}/P\qquad\mathbf{G_{B}}=\mathbf{H}\mathbf{B}\mathbf{B}^{T}\mathbf{H}^{T}/P. (14)

For the linear case, the HSIC is the dot product between the elements of the two second-moment matrices.

HSIC A , B = vec ​ ( 𝐆 𝐀 ) T ​ vec ​ ( 𝐆 𝐁 ) \mathrm{HSIC_{A,B}}=\mathrm{vec}(\mathbf{G_{A}})^{T}\mathrm{vec}(\mathbf{G_{B}}) (15)

The linear CKA is the normalized version of this quantity, just like the correlation coefficient is a normalized version of the covariance.

CKA = HSIC A , B HSIC A , A ​ HSIC B , B \mathrm{CKA}=\frac{\mathrm{HSIC}_{A,B}}{\sqrt{\mathrm{HSIC}_{A,A}\mathrm{HSIC}_{B,B}}} (16)

To show the equivalence of Eq. 11 (using biased distance estimates) and Eq. 16 , we can express the vector of squared distances 𝐝 𝐀 \mathbf{d_{A}} as a linear combination of the elements of the second moment matrix, d j \displaystyle d_{j} = \displaystyle= ( 𝐛 i − 𝐛 k ) ​ ( 𝐛 i − 𝐛 k ) T \displaystyle(\mathbf{b}_{i}-\mathbf{b}_{k})(\mathbf{b}_{i}-\mathbf{b}_{k})^{T} (17) = \displaystyle= 𝐆 i , i + 𝐆 k , k − 𝐆 i , k − 𝐆 k , i , \displaystyle\mathbf{G}_{i,i}+\mathbf{G}_{k,k}-\mathbf{G}_{i,k}-\mathbf{G}_{k,i}, (18)

which we can write more succinctly using a properly defined linear transformation matrix 𝐓 d \mathbf{T}_{d} , such that

𝐝 𝐀 = 𝐓 d ​ vec ​ ( 𝐆 𝐀 ) . \mathbf{d_{A}}=\mathbf{T}_{d}\mathrm{vec}(\mathbf{G_{A}}). (19)

The structure of the 𝐓 d ​ 𝐓 d T \mathbf{T}_{d}\mathbf{T}_{d}^{T} can be shown to be identical to that of the covariance matrix of the dissimilarities (Eq. 27 , 29 ), which is determined by the D × K D\times K contrast matrix 𝐂 \mathbf{C} (see methods):

vec ​ ( 𝐆 𝐀 ) T ​ vec ​ ( 𝐆 𝐁 ) = 𝐝 𝐀 T ​ ( 𝐓 d ​ 𝐓 d T ) − 1 ​ 𝐝 𝐁 = 𝐝 𝐀 T ​ ( 𝐂𝐂 T ∘ 𝐂𝐂 T ) − 1 ​ 𝐝 𝐁 = 𝐝 𝐀 T ​ ( c ​ 𝐕 ) − 1 ​ 𝐝 𝐁 . \begin{split}\mathrm{vec}(\mathbf{G_{A}})^{T}\mathrm{vec}(\mathbf{G_{B}})&=\mathbf{d_{A}}^{T}(\mathbf{T}_{d}\mathbf{T}_{d}^{T})^{-1}\mathbf{d_{B}}\\ &=\mathbf{d_{A}}^{T}(\mathbf{CC}^{T}\circ\mathbf{CC}^{T})^{-1}\mathbf{d_{B}}\\ &=\mathbf{d_{A}}^{T}(c\mathbf{V})^{-1}\mathbf{d_{B}}.\end{split} (20)

A more intuitive explanation of this equivalence is that all unique elements of an estimate of 𝐆 \mathbf{G} are mutually uncorrelated, if the true 𝐆 = 𝟎 \mathbf{G}=\mathbf{0} and 𝚺 K = 𝐈 ​ σ 2 \bm{\Sigma}_{K}=\mathbf{I}\sigma^{2} (Eq. 36 ). The covariance-structure of the distances is then simply induced by the fact that some of the distances share common elements of the second moment matrix, with the covariance structure determined (up to a constant) by 𝐓 d ​ 𝐓 d T \mathbf{T}_{d}\mathbf{T}_{d}^{T} .

While the whitened RDM cosine similarity using biased distance estimates is equivalent to the linear CKA or the RV coefficient, the whitened unbiased RDM cosine similarity (Eq. 5 , WUC) defines the unbiased version of this coefficient. For the special case of i.i.d. noise ( 𝚺 K = 𝐈 ​ σ 2 \bm{\Sigma}_{K}=\mathbf{I}\sigma^{2} ), it can be computed based on Eq. 15 and 16 , replacing 𝐆 \mathbf{G} with an unbiased estimate for the second moment matrix.

𝐆 ~ = 1 M ⁡ ( M − 1 ) ​ ∑ m ∑ n ≠ m 𝐇 ​ 𝐁 ^ m ​ 𝐁 ^ n T ​ 𝐇 T / P \tilde{\mathbf{G}}=\frac{1}{M(M-1)}\sum_{m}\sum_{n\neq m}{\mathbf{H}\hat{\mathbf{B}}_{m}\hat{\mathbf{B}}_{n}^{T}\mathbf{H}^{T}}/P (21)

For the more general case of arbitrary heteroskedasticity ( 𝚺 K ≠ 𝐈 ​ σ 2 \bm{\Sigma}_{K}\neq\mathbf{I}\sigma^{2} ), the WUC provides a way of incorporating a known noise covariance of the data for optimal inference. In the quadratic form in Eq. 20 , the term 𝐂𝐂 T ∘ 𝐂𝐂 T \mathbf{CC}^{T}\circ\mathbf{CC}^{T} can be replaced with 𝐂 ​ 𝚺 K ​ 𝐂 T ∘ 𝐂 ​ 𝚺 K ​ 𝐂 T \mathbf{C}\bm{\Sigma}_{K}\mathbf{C}^{T}\circ\mathbf{C}\bm{\Sigma}_{K}\mathbf{C}^{T} , ensuring that uneven measurement noise across conditions is being taken into account during model comparison.

## 3 Discussion

RSA provides an intuitive and flexible way of performing inference on representational models (i.e., on models that describe the relationship between high-dimensional activity patterns). There are, however, numerous different dissimilarity measures and ways of comparing measured RDMs to model RDMs, and the optimal way of implementing RSA remains a matter of debate [ 8 , 19 ] . In this paper, we derive an analytical expression for the mean and the covariance of the biased and unbiased estimates of squared Euclidean and Mahalanobis distances. In the following we will discuss the theoretical and practical aspects arising from these results.

### 3.1 Dealing with bias and covariance of distance estimators

First, we show that standard distance estimates are positively biased, and that this bias depends on the variances and covariances of the measured activity patterns ( 𝚺 K \bm{\Sigma}_{K} ). If the measurement noise is i.i.d. across trials, the bias will be the same across all distance estimates, and can be taken into account by ignoring the mean distance in subsequent model comparisons (for example by using the Pearson or rank correlation). If, however, one condition is measured with higher variance (for example because there were different numbers of repetitions or error trials had to be discarded), then all distances involving this condition will tend to be higher. If two conditions systematically follow each other, such that they are measured with a positive covariance, their dissimilarity will be systematically lower than two conditions measured with independent noise. These biases can translate into biases in model selection [ 10 ] . To avoid such errors, we can remove the bias in the estimation of the distance using crossvalidation. This approach has the substantial advantage that unequal measurement errors across conditions can no longer bias model decisions. However, removing the bias of the distance estimates comes at a cost: the variance of unbiased distance estimate is slightly higher than the variance of biased distance estimates, to be precise, by factor M / ( M − 1 ) M/(M-1) . Thus, when using unbiased estimates, a large number of independent partitions ( M M ) is desirable.

Second, we show that dissimilarity estimates within an RDM are systematically correlated with each other. In a previous paper, we had shown that model selection using the RDM cosine similarity or RDM correlation was less accurate than PCM [ 19 ] . Here we show that taking the covariance structure of the dissimilarity estimates into account improves our power to adjudicate between models. This improvement can even be achieved when using the covariance structure predicted under the assumption that all true distances are zero, which dramatically simplifies the procedure, avoiding iterative calculation of the covariance matrix. The power achieved with the whitened RDM cosine similarity (Eq. 11 ) or the whitened RDM Pearson correlation (Eq. 12 ) is close to the theoretical optimum, as achieved with the likelihood-ratio test implemented by PCM.

Taken together, these two insights suggest the use of unbiased distance estimates combined with the whitened cosine similarity to compare RDMs. We call this new approach the whitened unbiased RDM cosine similarity (WUC). It has important connections to the linear CKA [ 12 , 23 ] ) and RV coefficient [ 11 ] , but extends these two traditional approaches by removing the biasing influence of measurement noise by using a crossvalidated estimate of the distances (Eq. 5 ) or second moment matrix (Eq. 21 ).

### 3.2 When should the whitened unbiased RDM cosine similarity be used?