Comparing representational geometries using whitened unbiased-distance-matrix similarity β part 2/6
Cleaned-text code points [15310, 30887). Bibliography omitted here; source.md includes it.
Both expressions have two components: The first term of the equation arises from the multiplication of noise with noise. The variance of the distance estimate scales in the square of the measurement variance of the corresponding pattern differences ( π΅ k β k \bm{\Xi}_{kk} ). The only difference between the biased and unbiased estimate is the size of this component, which is larger by factor M / ( M β 1 ) M/(M-1) for the unbiased estimate. The second term arises from the multiplication of the true pattern difference ( πΉ k \bm{\delta}_{k} ) with measurement noise. If the true distance is zero, i.e. if there are no differences between the true activity patterns, this second term vanishes. The overall balance between the first and second term depends on the strength of the signal ( πΉ k \bm{\delta}_{k} ), and on the noise covariance structure across channels ( πΊ P \bm{\Sigma}_{P} ).
The insights from the equations are summarized in Figure 3, which shows the mean and variance of the squared distance estimates for a range of true distances between 0 and 1.2. If the true distance is 0, the mean of the unbiased estimate is zero, whereas the mean of the biased estimate is inflated by π΅ k β k / M \bm{\Xi}_{kk}/M . In exchange, the variance of the biased estimate is lower by a factor of ( M β 1 ) / M (M-1)/M . This difference is caused by using different numbers of pairwise products: The biased estimate uses all M 2 M^{2} possible pairs, whereas the unbiased estimate excludes M M of the pairs (those of each partition with itself). Thus, with M = 2 M=2 partitions, the variance of the unbiased estimate will be twice as large. However, the difference diminishes as the number of independent partitions increases. The second term of Eq. 7 , 8 causes the variance of the distance estimate to increase linearly with the true squared distance. This signal-dependence affects biased and unbiased estimates equally.
### 2.4 Model comparison using RDM correlations or cosine similarity
Whether it is better to use biased or unbiased distance estimates depends on how these estimates are used in subsequent inference. A common use case is to compare the measured RDM to different competing models of neural representations. For this, the upper triangular part of the RDM (which is symmetric about a diagonal of zeros) is vectorized (Fig. 1). The vector of estimated distances ( π ^ \hat{\mathbf{d}} or π ~ \tilde{\mathbf{d}} ) is then compared with the vector of model-predicted distances ( π¦ \mathbf{m} ), and the model with the best correspondence is selected.
Which measure of correspondence is appropriate depends on the level at which the models are meant to make predictions. If the models predict merely the rank-ordering of the distances, a Spearman rank correlation (or when any of the models predict equal representational distances for different pairs of stimuli, Kendallβs Ο A \tau_{A} rank correlation) is appropriate [ 8 ] . If the models make predictions about distances on an interval scale, one can use the Pearson correlation, r β‘ ( π ^ , π¦ CLOSE r(\hat{\mathbf{d}},\mathbf{m} ). Calculating a correlation (whether Kendall, Spearman, or Pearson) allows for arbitrary scaling between observed and predicted distances, which is useful because the model-predicted dissimilarities are typically in arbitrary units and the scaling of the data depends on the signal-to-noise ratio. These correlation coefficients are also invariant to an additive constant. For biased distance estimates this is useful, as it discounts a positive bias arising from noise. Note, however, that this only works if the noise influences all conditions equally, i.e. when all distances show the same constant bias.
In contrast, cross-validated distances estimates do not have any bias, no matter whether it is varies or is constant across conditions. The resultant distances have a ratio-scale measure, where 0 is an informative point, indicating that the two patterns only differ by measurement noise. To exploit this additional information, we can compute the correlation without subtracting the mean across distances. This quantity is the RDM cosine similarity, the cosine of the angle between the vectorized data RDM and the model RDM (Fig. 2).
r ~ = π ~ T β π¦ ( π ~ T β π ~ ) β ( π¦ T β π¦ ) \tilde{r}=\frac{\tilde{\mathbf{d}}^{T}\mathbf{m}}{\sqrt{(\tilde{\mathbf{d}}^{T}\tilde{\mathbf{d}})(\mathbf{m}^{T}\mathbf{m})}} (9)
In summary, for models that predict the RDM on a ratio-scale we have two consistent choices: We can either compute the cosine similarity between the model and unbiased distance estimates (Eq. 9 , in short RDM cosine similarity ), or the Pearson correlation between the model and biased distance estimates (in short RDM correlation ).
### 2.5 Using RDM cosine similarity prevents bias in model selection
RDM correlations will only account for the biasing influence of measurement noise if the positive bias is the same across all distance estimates in the RDM - that is, when all pairwise pattern differences are measured with the same variability. If one condition has a smaller number of trials than another (e.g., after the exclusion of error trials), distances involving this condition will be systematically larger (Eq. 4 ). Similarly, if the measurement errors for one pair of conditions are more correlated than for another pair (e.g. because some conditions were measured with fMRI in temporal proximity), then the variance of their pattern differences will be smaller, and the distance estimates lower. In both cases, the use of RDM correlations can bias inference towards the incorrect model [ 10 ] .
Consider for example the two RSA models depicted in Fig. 4a. In the first model, condition 1 and 2 belong to one category, and conditions 3 and 4 belong to another category. Within-category distances are predicted to be smaller than between-category distances. In the second model, condition 1 and 3, and condition 2 and 4 belong to the same category. We simulated pure noise data, for which the measurement noise was slightly correlated (r=0.15) across neighboring conditions (i.e. between 1-2, 2-3, and 3-4). Such correlation can for example occur in fMRI experiments, when the conditions were collected in a fixed sequence, such as in a βtraveling waveβ design, often used for perceptual fMRI experiments [ 17 , 18 ] . When we used RDM correlations to compare the data to the two models, 76.8% of simulations were attributed to model 1 and only 23.6% to model 2. This bias arises because the measurement noise induces a similarity structure (lower distances between neighbouring conditions) that is more similar to model 1 than model 2. When comparing models using unbiased distance estimates, the model selection bias disappeared - now exactly 50% of the null simulations were assigned to each model.
To study the influence of this bias on model discrimination, we simulated datasets coming from each model, while varying the number of independent partitions. We again added correlated noise to each partition, and counted how often a dataset was assigned to the correct model. Decisions using RDM correlations were on average 65.1% correct (Fig. 4a, blue line), independent of the number of partitions (note that we left the overall amount of data the same, see methods). In contrast, decisions using RDM cosine similarity improved with increasing number of partitions (Fig. 4a, red line). For 2 partitions, decisions based on RDM cosine similarity were less often correct than those based on RDM correlations, but for more than 2 partitions, decisions based on crossvalidated distance estimates performed better. This increase arises because the variance for the unbiased estimates depends on factor M / ( M β 1 ) M/(M-1) .
When we repeated the same simulation, using independent noise for each condition, the bias disappeared, and so did the disadvantage in model selection accuracy (Fig. 4b). Now decisions based on RDM correlations were always more accurate, although the difference became smaller with an increasing number of partitions.
### 2.6 RDM cosine similarity exploits informative zero-point
Does this mean that we should use RDM correlations with biased distances estimates when the measurement noise is i.i.d.? Not necessarily: Using the RDM cosine similarity can have advantages for some model comparisons, as it exploits the additional information inherent in the zero point. Consider the two models depicted in Fig. 4c. Both models have the same category structure, but differ in the predicted ratio of the within-category to the between-category distances. To distinguish these models, a meaningful zero point is required. When calculating the correlation between two RDMs, the mean across all distances in each RDM is first subtracted. Thus, correlations remove any constant offset of the distances in model and data - and model selection therefore remains at chance level (0.5). Inferences can only be made using the RDM cosine similarities on unbiased distance estimates.
Finally, there are also cases in which the model decision is not fully dependent on the zero point (as in Fig. 4c), but also not fully independent of it (as in Fig. 4b). For example, the two models in in Fig. 4d have a different category structure, but also differ in the size of the distances relative to zero. As a result, we observe that RDM cosine similarities just outperform RDM correlations for larger number of partitions. Note that decisions on Null-data are unbiased in either case, as the measurement noise was modeled as i.i.d.
In summary, how the larger variability of crossvalidated distance estimates translates into model selection accuracy, depends both on the number of available partitions, as well as the structure of the two models that are being compared. In any case, unbiased distance estimates provide a safeguard against deviations from the assumption of i.i.d measurement noise β an advantage that in many cases will be well worth the small cost in statistical power.
### 2.7 Covariance of distance estimates
The use of RDM correlations (for biased distance estimates) or RDM cosine similarity (for unbiased estimates) for model comparison would be fully adequate if all elements of the RDM were estimated independently and with the same variance. However, our analytical expression for the full covariance matrix of dissimilarities (Eq. 27 , 29 ) shows that this is not the case, even if the underlying activity patterns are measured with i.i.d. noise.
The correlation between distance estimates arises from the fact that, even if all conditions are measured with i.i.d. noise, the pattern difference between conditions 1 and 2 is not independent from the pattern difference between conditions 1 and 3. The covariance matrix for the 10 distance for a design with 5 conditions (assuming no true pattern differences and i.i.d. noise) is shown in Figure 5a. Distances that share one of the conditions (i.e., between d 1 , 2 d_{1,2} and d 1 , 3 d_{1,3} ) have a correlation of r = 0.25 r=0.25 . Only estimates of distances that do not share any conditions (i.e., d 1 , 2 d_{1,2} and d 3 , 4 d_{3,4} ) are uncorrelated. If the measurement error on the patterns is not i.i.d., a more complex co-dependence structure can arise.
For a design with a larger number of conditions (K=18, Fig. 2b), the number of uncorrelated distances increases, i.e. the covariance matrix becomes sparser. This does not mean, however, that we can ignore the dependence structure. As we will show below, accounting for the co-dependence structure becomes especially important in designs with large numbers of conditions.
Similar to what we have observed for the variance, the covariance between distance estimates also increases with increasing true distances. Figure 5c shows the covariance matrix for distance estimates from data generated from the RDM model depicted in Figure 1. Larger true distances exhibit larger variances and also larger covariances with other distances. The exact shape of the covariance matrix depends on the exact form of the true pattern differences πΉ \bm{\delta} , as well as the covariance of noise across voxels ( πΊ P \bm{\Sigma}_{P} ).
Given the correlation between elements of the estimated RDM, we therefore would expect that both RDM correlations ( r r ) and RDM cosine angles ( r ~ \tilde{r} ) will perform sub-optimally (see Figure 2). Indeed, in a previous paper [ 19 ] , we have shown, for several different simulation scenarios, that ignoring this covariance structure leads to 3-12% fewer correct model-selection decisions, as compared to a full likelihood-ratio test between models, implemented in Pattern Component Modelling [ 14 , 19 ] .
### 2.8 RDM comparison in whitened RDM-space
To improve model inference, we should therefore use the covariance matrix to transform the elements of the RDM into a whitened space, in which all dimensions are measured independently and with the same variance. Originally [ 19 ] , we had suggested using the full expression for the covariance (Eq. 29 ), and to estimate required quantities from the data. This approach, however, can be slow, as the estimation must be done iteratively, which each step involving the inversion of a K β‘ ( K β 1 ) Γ K β‘ ( K β 1 ) K(K-1)\times K(K-1) matrix - i.e. the complexity of the algorithm scales with πͺ β‘ ( K 4 ) \mathcal{O}(K^{4}) . Furthermore, for fMRI data, the spatial structure of the signal ( πΉ \bm{\delta} ) and the measurement noise ( πΊ P \bm{\Sigma}_{P} ) are hard to estimate separately.
We therefore propose a simplification to avoid estimating the right side of Eq. 27 and Eq. 29 . Specifically, we suggest using the covariance structure of the distances under the assumption that all distances are zero. In this case the variance simplifies to
Var β‘ ( π ~ ) = c β‘ ( π΅ β π΅ ) = c β π , \begin{split}\mathrm{Var}(\tilde{\mathbf{d}})&=c(\bm{\Xi}\circ\bm{\Xi})=c\mathbf{V},\end{split} (10)
where π \mathbf{V} is the structure of the covariance matrix, and c c is a proportionality constant that is not important for most applications. Note that noise correlations between different channels (or voxels in fMRI) would only influence c c , but not π \mathbf{V} (see Discussion). The matrix π΅ \bm{\Xi} can be readily calculated from an estimate of πΊ K \bm{\Sigma}_{K} (see Eq. 23 ). To take this covariance matrix into account for model comparison, we can prewhiten the distances by pre-multiplying them with π β 1 2 \mathbf{V}^{-\frac{1}{2}} .
In the case of the cosine similarity between unbiased distance estimates, this leads to a new criterion, the whitened unbiased RDM cosine similarity (WUC):
r ~ w = π ~ T β π β 1 β π¦ ( π ~ T β π β 1 β π ~ ) β ( π¦ T β π β 1 β π¦ ) \tilde{r}_{w}=\frac{\tilde{\mathbf{d}}^{T}\mathbf{V}^{-1}\mathbf{m}}{\sqrt{(\tilde{\mathbf{d}}^{T}\mathbf{V}^{-1}\tilde{\mathbf{d}})(\mathbf{m}^{T}\mathbf{V}^{-1}\mathbf{m})}} (11)
Similarly, we can define a whitened RDM Pearson correlation, simply by first subtracting the mean of data ( π Β― \bar{\mathbf{d}} ) and model ( π¦ Β― \bar{\mathbf{m}} ) RDM.
r w = ( π ^ β π Β― ) T β π β 1 β ( π¦ β π¦ Β― ) ( ( π ^ β π Β― ) T β π β 1 β ( π ^ β π Β― ) ) β ( ( π¦ β π¦ Β― ) T β π β 1 β ( π¦ β π¦ Β― ) ) r_{w}=\frac{(\hat{\mathbf{d}}-\bar{\mathbf{d}})^{T}\mathbf{V}^{-1}(\mathbf{m}-\bar{\mathbf{m}})}{\sqrt{((\hat{\mathbf{d}}-\bar{\mathbf{d}})^{T}\mathbf{V}^{-1}(\hat{\mathbf{d}}-\bar{\mathbf{d}}))((\mathbf{m}-\bar{\mathbf{m}})^{T}\mathbf{V}^{-1}(\mathbf{m}-\bar{\mathbf{m}}))}} (12)
### 2.9 Influence of RDM whitening on model comparisons