Estimating Neural Representation Alignment from Sparsely Sampled Inputs and Features — part 4/5

Cleaned-text code points [47125, 63020). Bibliography omitted here; source.md includes it.

v i ​ j ​ l ​ m = K i ​ j ( a ) ​ K l ​ m ( a ) = 1 Q 2 ​ ∑ α ​ β Φ i ​ α ( a ) ​ Φ j ​ α ( a ) ​ Φ l ​ β ( a ) ​ Φ m ​ β ( a ) v_{ijlm}=K_{ij}^{(a)}K_{lm}^{(a)}=\frac{1}{Q^{2}}\sum_{\alpha\beta}\Phi_{i\alpha}^{(a)}\Phi_{j\alpha}^{(a)}\Phi_{l\beta}^{(a)}\Phi_{m\beta}^{(a)} If we were to take the expected value of v i ​ j ​ l ​ m v_{ijlm} over the feature sampling, we need to consider the overlapping of the feature indices α \alpha and β \beta . Therefore, the following should be an unbiased estimator of v i ​ j ​ l ​ m v_{ijlm} :

c i ​ j ​ l ​ m ′ = 1 Q ⁡ ( Q − 1 ) ​ ∑ α ≠ β Φ i ​ α ( a ) ​ Φ j ​ α ( a ) ​ Φ l ​ β ( a ) ​ Φ m ​ β ( a ) = 1 Q ⁡ ( Q − 1 ) ​ ( ∑ α ​ β Φ i ​ α ( a ) ​ Φ j ​ α ( a ) ​ Φ l ​ β ( a ) ​ Φ m ​ β ( a ) − ∑ α Φ i ​ α ( a ) ​ Φ j ​ α ( a ) ​ Φ l ​ α ( a ) ​ Φ m ​ α ( a ) ) . c^{\prime}_{ijlm}=\frac{1}{Q\left(Q-1\right)}\sum_{\alpha\neq\beta}\Phi_{i\alpha}^{(a)}\Phi_{j\alpha}^{(a)}\Phi_{l\beta}^{(a)}\Phi_{m\beta}^{(a)}=\frac{1}{Q\left(Q-1\right)}\left(\sum_{\alpha\beta}\Phi_{i\alpha}^{(a)}\Phi_{j\alpha}^{(a)}\Phi_{l\beta}^{(a)}\Phi_{m\beta}^{(a)}-\sum_{\alpha}\Phi_{i\alpha}^{(a)}\Phi_{j\alpha}^{(a)}\Phi_{l\alpha}^{(a)}\Phi_{m\alpha}^{(a)}\right). In the main text we define c i ​ j ​ l ​ m c_{ijlm} , which is simply c i ​ j ​ l ​ m ′ c^{\prime}_{ijlm} without the 1 Q ⁡ ( Q − 1 ) \frac{1}{Q\left(Q-1\right)} factor, i.e. c i ​ j ​ l ​ m = Q ⁡ ( Q − 1 ) ​ c i ​ j ​ l ​ m ′ c_{ijlm}=Q\left(Q-1\right)c^{\prime}_{ijlm} . Therefore, finally, our estimator can be expressed as

ℋ ^ C ( Φ ( a ) , Φ ( a ) ) = 1 P 3 ​ ( P − 3 ) ​ Q ​ ( Q − 1 ) × ∑ i ​ j ​ l ​ m ( ( c i ​ j ​ i ​ j − c i ​ i ​ i ​ i P ) − 2 ​ P P − 2 ​ ( c i ​ j ​ j ​ l − c i ​ i ​ i ​ j P − c i ​ j ​ j ​ j P + c i ​ i ​ i ​ i P 2 ) + P 2 ( P − 1 ) ​ ( P − 2 ) ​ ( c i ​ j ​ l ​ m − c i ​ i ​ j ​ l P − c j ​ l ​ i ​ i P + c i ​ i ​ j ​ j P 2 ) ) \widehat{\mathcal{H}}_{C}(\Phi^{(a)},\Phi^{(a)})=\frac{1}{P^{3}\left(P-3\right)Q\left(Q-1\right)}\times\\ \sum_{ijlm}\left(\left(c_{ijij}-\frac{c_{iiii}}{P}\right)-\frac{2P}{P-2}\left(c_{ijjl}-\frac{c_{iiij}}{P}-\frac{c_{ijjj}}{P}+\frac{c_{iiii}}{P^{2}}\right)+\frac{P^{2}}{\left(P-1\right)\left(P-2\right)}\left(c_{ijlm}-\frac{c_{iijl}}{P}-\frac{c_{jlii}}{P}+\frac{c_{iijj}}{P^{2}}\right)\right) (S3)

## Appendix B Bias analysis

### B.1 Generative process framework

Here we formalize the problem setup by defining a general formulation of the process that generates the measurement matrices. Consider a pair of measurement matrices Φ ( a ) \Phi_{(a)} and Φ ( b ) \Phi_{(b)} of two systems a a and b b . Let x i ∈ 𝒳 x_{i}\in\mathcal{X} be latent variables for the i i th rows of Φ ( a ) \Phi_{(a)} and Φ ( b ) \Phi_{(b)} , and u α ∈ 𝒰 u_{\alpha}\in\mathcal{U} and v α ∈ 𝒱 v_{\alpha}\in\mathcal{V} be column-latent variables for the α \alpha th column of Φ ( a ) \Phi_{(a)} , and Φ ( b ) \Phi_{(b)} , respectively. Let P P be the number of row-latent variables sampled, and Q a Q_{a} and Q b Q_{b} be the numbers of column-latent variables sampled. Let ϕ a : 𝒳 × 𝒰 → ℝ \phi_{a}\mathrel{\mathop{\mathchar 58\relax}}\mathcal{X}\times\mathcal{U}\to\mathbb{R} be a map that defines the measurement value. Then we assume the entries of Φ ( a ) ∈ ℝ P × Q a \Phi_{(a)}\in\mathbb{R}^{P\times Q_{a}} and Φ ( b ) ∈ ℝ P × Q b \Phi_{(b)}\in\mathbb{R}^{P\times Q_{b}} are defined as Φ i ​ α ( a ) = ϕ a ​ ( x i , u α ) , and ​ Φ i ​ β ( b ) = ϕ b ​ ( x i , v β ) . \Phi_{i\alpha}^{(a)}=\phi_{a}(x_{i},u_{\alpha}),\text{and }\Phi_{i\beta}^{(b)}=\phi_{b}(x_{i},v_{\beta}). In some cases, a latent space might be fully observed in the measurement. For example, N N dimensional layer of neural network activation is given by a feature map ψ ( a ) : 𝒳 → ℝ N \psi_{(a)}\mathrel{\mathop{\mathchar 58\relax}}\mathcal{X}\to\mathbb{R}^{N} , where 𝒳 \mathcal{X} is the input space, and the column latent set 𝒰 \mathcal{U} of cardinality N N is fully observed. Here, 𝒰 \mathcal{U} would be a set of trained neural network weights. The uncentered kernel would be simply k ′ ​ ( x , x ′ ) = 1 N ​ ψ ​ ( x ) ​ ψ ​ ( x ′ ) ⊤ k^{\prime}(x,x^{\prime})=\frac{1}{N}\psi(x)\psi(x^{\prime})^{\top} . If the latent variables cannot be fully observed, we assume there are probability measures over the latent spaces: ρ 𝒳 \rho_{\mathcal{X}} is the probability measures over 𝒳 \mathcal{X} and, and similarly for ρ 𝒰 \rho_{\mathcal{U}} and ρ 𝒱 \rho_{\mathcal{V}} . Assume ϕ a \phi_{a} is square integrable w.r.t. ρ 𝒳 \rho_{\mathcal{X}} and ρ 𝒰 \rho_{\mathcal{U}} , and similarly ϕ b \phi_{b} is also square integrable. The associated uncentered kernels are defined as k a ′ ​ ( x , x ′ ) = ∫ d ​ ρ 𝒰 ​ ( u ) ​ ϕ a ​ ( x , u ) ​ ϕ a ​ ( x ′ , u ) k^{\prime}_{a}(x,x^{\prime})=\int d\rho_{\mathcal{U}}(u)\>\phi_{a}(x,u)\phi_{a}(x^{\prime},u) and k b ′ ​ ( y , y ′ ) = ∫ d ​ ρ 𝒱 ​ ( v ) ​ ϕ b ​ ( y , v ) ​ ϕ b ​ ( y ′ , v ) k^{\prime}_{b}(y,y^{\prime})=\int d\rho_{\mathcal{V}}(v)\>\phi_{b}(y,v)\phi_{b}(y^{\prime},v) . We may also define the associated kernel integral operator:

T k ​ f = ∫ d ​ ρ 𝒳 ​ ( x ) ​ k ′ ​ ( ⋅ , x ) ​ f ​ ( x ) . T_{k}f=\int d\rho_{\mathcal{X}}(x)\,k^{\prime}(\cdot,x)f(x). (S4) Later, the eigenvalues of the operator T k T_{k} will be relevant.

We also define associated covariance kernels: k ~ a ′ ​ ( u , u ′ ) = ∫ d ​ ρ 𝒳 ​ ( x ) ​ ϕ a ​ ( x , u ) ​ ϕ a ​ ( x , u ′ ) \tilde{k}^{\prime}_{a}(u,u^{\prime})=\int d\rho_{\mathcal{X}}(x)\>\phi_{a}(x,u)\phi_{a}(x,u^{\prime}) and k ~ b ′ ​ ( v , v ′ ) = ∫ d ​ ρ 𝒳 ​ ( x ) ​ ϕ b ​ ( x , v ) ​ ϕ b ​ ( x , v ′ ) \tilde{k}^{\prime}_{b}(v,v^{\prime})=\int d\rho_{\mathcal{X}}(x)\>\phi_{b}(x,v)\phi_{b}(x,v^{\prime}) .

### B.2 Biases in CKA estimators contributed by the biases in HSIC estimators

Here we derive the analytical expression of the biases of the CKA estimators. Here we assume that each feature is already centered, i.e. ∫ d ​ ρ 𝒳 ​ ( x ) ​ ϕ a ​ ( x , ⋅ ) = 0 \int d\rho_{\mathcal{X}}(x)\>\phi_{a}(x,\cdot)=0 , and ∫ d ​ ρ 𝒳 ​ ( x ) ​ ϕ b ​ ( x , ⋅ ) = 0 \int d\rho_{\mathcal{X}}(x)\>\phi_{b}(x,\cdot)=0 . We first compute the expected values of the HSIC estimators. Let 𝒮 𝒳 ≔ { x i } i = 1 P \mathcal{S}_{\mathcal{X}}\coloneqq\left\{x_{i}\right\}_{i=1}^{P} , ℱ 𝒰 ≔ { u i } i = 1 Q a \mathcal{F}_{\mathcal{U}}\coloneqq\left\{u_{i}\right\}_{i=1}^{Q_{a}} , and ℱ 𝒱 ≔ { v i } i = 1 Q b \mathcal{F}_{\mathcal{V}}\coloneqq\left\{v_{i}\right\}_{i=1}^{Q_{b}} be the sets of latent variables sampled independently from ρ 𝒳 , \rho_{\mathcal{X}}, ρ 𝒰 \rho_{\mathcal{U}} , and ρ 𝒱 \rho_{\mathcal{V}} respectively. The naive HSIC estimator is then given by

ℋ 0 ^ ​ ( ϕ a , ϕ b , 𝒮 𝒳 , ℱ 𝒰 , ℱ 𝒱 ) = 1 P 2 ​ Q a ​ Q b ​ ∑ i ​ j ∑ α ∑ β ϕ a ​ ( x i , u α ) ​ ϕ a ​ ( x j , u α ) ​ ϕ b ​ ( x i , v β ) ​ ϕ b ​ ( x j , v β ) \widehat{\mathcal{H}_{0}}(\phi_{a},\phi_{b},\mathcal{S}_{\mathcal{X}},\mathcal{F}_{\mathcal{U}},\mathcal{F}_{\mathcal{V}})=\frac{1}{P^{2}Q_{a}Q_{b}}\sum_{ij}\sum_{\alpha}\sum_{\beta}\phi_{a}(x_{i},u_{\alpha})\phi_{a}(x_{j},u_{\alpha})\phi_{b}(x_{i},v_{\beta})\phi_{b}(x_{j},v_{\beta}) if u α u_{\alpha} and v β v_{\beta} are independently sampled for all α \alpha and β \beta combinations, which corresponds to the numerator of CKA. If the column latent variables are identical, and ϕ a = ϕ b \phi_{a}=\phi_{b} , this corresponds to the HSICs in the denominator of naive CKA: ℋ 0 ^ ​ ( ϕ a , 𝒮 𝒳 , ℱ 𝒰 ) = 1 P 2 ​ Q a 2 ​ ∑ i ​ j ∑ α ​ β ϕ a ​ ( x i , u α ) ​ ϕ a ​ ( x j , u α ) ​ ϕ a ​ ( x i , u β ) ​ ϕ a ​ ( x j , u β ) , \widehat{\mathcal{H}_{0}}(\phi_{a},\mathcal{S}_{\mathcal{X}},\mathcal{F}_{\mathcal{U}})=\frac{1}{P^{2}Q_{a}^{2}}\sum_{ij}\sum_{\alpha\beta}\phi_{a}(x_{i},u_{\alpha})\phi_{a}(x_{j},u_{\alpha})\phi_{a}(x_{i},u_{\beta})\phi_{a}(x_{j},u_{\beta}), ℋ 0 ^ ​ ( ϕ b , 𝒮 𝒳 , ℱ 𝒱 ) = 1 P 2 ​ Q b 2 ​ ∑ i ​ j ∑ α ​ β ϕ b ​ ( x i , v α ) ​ ϕ b ​ ( x j , v α ) ​ ϕ b ​ ( x i , v β ) ​ ϕ b ​ ( x j , v β ) \widehat{\mathcal{H}_{0}}(\phi_{b},\mathcal{S}_{\mathcal{X}},\mathcal{F}_{\mathcal{V}})=\frac{1}{P^{2}Q_{b}^{2}}\sum_{ij}\sum_{\alpha\beta}\phi_{b}(x_{i},v_{\alpha})\phi_{b}(x_{j},v_{\alpha})\phi_{b}(x_{i},v_{\beta})\phi_{b}(x_{j},v_{\beta})

Here, we assume the naive CKA is computed after N N trials, across which the empirical means of the HSIC are computed::

CKA ^ 0 ​ ( ϕ a , ϕ b , ρ 𝒳 , ρ 𝒰 , ρ 𝒱 ) = ∑ l = 1 N ℋ 0 ^ ​ ( ϕ a , ϕ b , 𝒮 𝒳 ( l ) , ℱ 𝒰 ( l ) , ℱ 𝒱 ( l ) ) ∑ l = 1 N ℋ 0 ^ ​ ( ϕ a , 𝒮 𝒳 ( l ) , ℱ 𝒰 ( l ) ) ​ ∑ l = 1 N ℋ 0 ^ ​ ( ϕ b , 𝒮 𝒳 ( l ) , ℱ 𝒱 ( l ) ) \widehat{\text{CKA}}_{0}(\phi_{a},\phi_{b},\rho_{\mathcal{X}},\rho_{\mathcal{U}},\rho_{\mathcal{V}})=\frac{\sum_{l=1}^{N}\widehat{\mathcal{H}_{0}}(\phi_{a},\phi_{b},\mathcal{S}_{\mathcal{X}}^{(l)},\mathcal{F}_{\mathcal{U}}^{(l)},\mathcal{F}_{\mathcal{V}}^{(l)})}{\sqrt{\sum_{l=1}^{N}\widehat{\mathcal{H}_{0}}(\phi_{a},\mathcal{S}_{\mathcal{X}}^{(l)},\mathcal{F}_{\mathcal{U}}^{(l)})\sum_{l=1}^{N}\widehat{\mathcal{H}_{0}}(\phi_{b},\mathcal{S}_{\mathcal{X}}^{(l)},\mathcal{F}_{\mathcal{V}}^{(l)})}} We assume N N is large such that all three ℋ ^ \widehat{\mathcal{H}} empirical averages have small variance of order 𝒪 ⁡ ( 1 N ) \mathcal{O}\left(\frac{1}{N}\right) . In this limit, the following approximation is valid:

⟨ CKA ^ 0 ​ ( ϕ a , ϕ b , 𝒮 𝒳 , ℱ 𝒰 , ℱ 𝒱 ) ⟩ ≈ ⟨ ℋ 0 ^ ​ ( ϕ a , ϕ b , 𝒮 𝒳 ( l ) , ℱ 𝒰 ( l ) , ℱ 𝒱 ( l ) ) ⟩ ⟨ ℋ 0 ^ ​ ( ϕ a , 𝒮 𝒳 ( l ) , ℱ 𝒰 ( l ) ) ⟩ ​ ⟨ ℋ 0 ^ ​ ( ϕ b , 𝒮 𝒳 ( l ) , ℱ 𝒱 ( l ) ) ⟩ . \left\langle\widehat{\text{CKA}}_{0}(\phi_{a},\phi_{b},\mathcal{S}_{\mathcal{X}},\mathcal{F}_{\mathcal{U}},\mathcal{F}_{\mathcal{V}})\right\rangle\approx\frac{\left\langle\widehat{\mathcal{H}_{0}}(\phi_{a},\phi_{b},\mathcal{S}_{\mathcal{X}}^{(l)},\mathcal{F}_{\mathcal{U}}^{(l)},\mathcal{F}_{\mathcal{V}}^{(l)})\right\rangle}{\sqrt{\left\langle\widehat{\mathcal{H}_{0}}(\phi_{a},\mathcal{S}_{\mathcal{X}}^{(l)},\mathcal{F}_{\mathcal{U}}^{(l)})\right\rangle\left\langle\widehat{\mathcal{H}_{0}}(\phi_{b},\mathcal{S}_{\mathcal{X}}^{(l)},\mathcal{F}_{\mathcal{V}}^{(l)})\right\rangle}}. This approximation is also valid when P P , Q a Q_{a} , and Q b Q_{b} are all large. We empirically observe that the expected value of CKA ^ 0 \widehat{\text{CKA}}_{0} obtained via this approximation still accurately predict the expected values of CKA ^ 0 \widehat{\text{CKA}}_{0} empirically computed even with N = 1 N=1 and often even in addition to small P P , Q a Q_{a} , and Q b Q_{b} . This allows us to understand the bias of CKA ^ 0 \widehat{\text{CKA}}_{0} contributed from the biases of ℋ ^ 0 \widehat{\mathcal{H}}_{0} , isolated from the bias contributed by taking products, inverse, and square root of the ℋ ^ 0 \widehat{\mathcal{H}}_{0} estimates, which unnecessarily complicates the analysis.

First, let us take the expected value of the ℋ ^ 0 \widehat{\mathcal{H}}_{0} in the numerator: ⟨ ℋ 0 ^ ​ ( ϕ a , ϕ b , 𝒮 𝒳 , ℱ 𝒰 , ℱ 𝒱 ) ⟩ = \displaystyle\left\langle\widehat{\mathcal{H}_{0}}(\phi_{a},\phi_{b},\mathcal{S}_{\mathcal{X}},\mathcal{F}_{\mathcal{U}},\mathcal{F}_{\mathcal{V}})\right\rangle= 1 P 2 ​ Q a ​ Q b ​ ∑ i ​ j ∑ α ∑ β ⟨ ϕ a ​ ( x i , u α ) ​ ϕ a ​ ( x j , u α ) ​ ϕ b ​ ( x i , v β ) ​ ϕ b ​ ( x j , v β ) ⟩ \displaystyle\frac{1}{P^{2}Q_{a}Q_{b}}\sum_{ij}\sum_{\alpha}\sum_{\beta}\left\langle\phi_{a}(x_{i},u_{\alpha})\phi_{a}(x_{j},u_{\alpha})\phi_{b}(x_{i},v_{\beta})\phi_{b}(x_{j},v_{\beta})\right\rangle (S5) = \displaystyle= ⟨ k a ​ ( x , y ) ​ k b ​ ( x , y ) ⟩ x , y + 1 P ​ ( ⟨ k a ​ ( x , x ) ​ k b ​ ( x , x ) ⟩ x − ⟨ k a ​ ( x , y ) ​ k b ​ ( x , y ) ⟩ x , y ) \displaystyle\left\langle k_{a}(x,y)k_{b}(x,y)\right\rangle_{x,y}+\frac{1}{P}\left(\left\langle k_{a}(x,x)k_{b}(x,x)\right\rangle_{x}-\left\langle k_{a}(x,y)k_{b}(x,y)\right\rangle_{x,y}\right) (S6) = \displaystyle= ⟨ k a ​ ( x , y ) ​ k b ​ ( x , y ) ⟩ x , y ​ ( 1 + 1 P ​ ( ζ a ​ b − 1 ) ) \displaystyle\left\langle k_{a}(x,y)k_{b}(x,y)\right\rangle_{x,y}\left(1+\frac{1}{P}\left(\zeta_{ab}-1\right)\right) (S7)

where we have introduced a new variable ζ a ​ b ≔ ⟨ k a ​ ( x , x ) ​ k b ​ ( x , x ) ⟩ x ⟨ k a ​ ( x , y ) ​ k b ​ ( x , y ) ⟩ x , y \zeta_{ab}\coloneqq\frac{\left\langle k_{a}(x,x)k_{b}(x,x)\right\rangle_{x}}{\left\langle k_{a}(x,y)k_{b}(x,y)\right\rangle_{x,y}} that is sensitive to the alignment of the representation. Note that the expected value of the stimulus-corrected estimator ℋ S ^ ​ ( ϕ a , ϕ b , 𝒮 𝒳 , ℱ 𝒰 , ℱ 𝒱 ) \widehat{\mathcal{H}_{S}}(\phi_{a},\phi_{b},\mathcal{S}_{\mathcal{X}},\mathcal{F}_{\mathcal{U}},\mathcal{F}_{\mathcal{V}}) can be obtained by taking P → ∞ P\to\infty limit in Equation S7 , which simply ⟨ k a ​ ( x , y ) ​ k b ​ ( x , y ) ⟩ \left\langle k_{a}(x,y)k_{b}(x,y)\right\rangle , which means that the numerator ℋ S ^ \widehat{\mathcal{H}_{S}} is unbiased in this problem setup. However, in an alternative problem setup, such as the trial-to-trial CKA, the numerator ℋ S ^ \widehat{\mathcal{H}_{S}} is still biased, since the neurons are identical across (a) and (b), i.e. u α = v α u_{\alpha}=v_{\alpha} . The bias of the numerator ℋ S ^ \widehat{\mathcal{H}_{S}} in this alternative problem setup is similar to the biases in the denominator ℋ S ^ \widehat{\mathcal{H}_{S}} ’s of both problem setup.

Next, let us take the expected value of one of the ℋ ^ 0 \widehat{\mathcal{H}}_{0} ’s in the denominator:

⟨ ℋ 0 ^ ​ ( ϕ a , 𝒮 𝒳 , ℱ 𝒰 ) ⟩ = 1 P 2 ​ Q a 2 ​ ∑ i ​ j ∑ α ​ β ⟨ ϕ a ​ ( x i , u α ) ​ ϕ a ​ ( x j , u α ) ​ ϕ a ​ ( x i , u β ) ​ ϕ a ​ ( x j , u β ) ⟩ \left\langle\widehat{\mathcal{H}_{0}}(\phi_{a},\mathcal{S}_{\mathcal{X}},\mathcal{F}_{\mathcal{U}})\right\rangle=\frac{1}{P^{2}Q_{a}^{2}}\sum_{ij}\sum_{\alpha\beta}\left\langle\phi_{a}(x_{i},u_{\alpha})\phi_{a}(x_{j},u_{\alpha})\phi_{a}(x_{i},u_{\beta})\phi_{a}(x_{j},u_{\beta})\right\rangle

= ⟨ k a ​ ( x , y ) 2 ⟩ − 1 P ​ ( ⟨ k a ​ ( x , y ) 2 ⟩ − ⟨ k a ​ ( x , x ) 2 ⟩ ) − 1 Q a ​ ( ⟨ k a ​ ( x , y ) 2 ⟩ − ⟨ k ~ a ​ ( w , w ) 2 ⟩ ) + 1 P ​ Q a ​ ( ⟨ k a ​ ( x , y ) 2 ⟩ − ⟨ k a ​ ( x , x ) 2 ⟩ − ⟨ k ~ a ​ ( w , w ) 2 ⟩ + ⟨ ϕ a ​ ( x , w ) 4 ⟩ ) =\left\langle k_{a}(x,y)^{2}\right\rangle-\frac{1}{P}\left(\left\langle k_{a}(x,y)^{2}\right\rangle-\left\langle k_{a}(x,x)^{2}\right\rangle\right)-\frac{1}{Q_{a}}\left(\left\langle k_{a}(x,y)^{2}\right\rangle-\left\langle\tilde{k}_{a}(w,w)^{2}\right\rangle\right)+\\ \frac{1}{PQ_{a}}\left(\left\langle k_{a}(x,y)^{2}\right\rangle-\left\langle k_{a}(x,x)^{2}\right\rangle-\left\langle\tilde{k}_{a}(w,w)^{2}\right\rangle+\left\langle\phi_{a}(x,w)^{4}\right\rangle\right) (S8)

= ⟨ k a ​ ( x , y ) 2 ⟩ ​ [ 1 + 1 P ​ ( γ a ψ a − 1 ) + 1 Q a ​ ( γ a ψ ~ a − 1 ) − 1 P ​ Q a ​ ( γ a ψ a + γ a ψ ~ a − γ a ρ a − 1 ) ] =\left\langle k_{a}(x,y)^{2}\right\rangle\left[1+\frac{1}{P}\left(\frac{\gamma_{a}}{\psi_{a}}-1\right)+\frac{1}{Q_{a}}\left(\frac{\gamma_{a}}{\tilde{\psi}_{a}}-1\right)-\frac{1}{PQ_{a}}\left(\frac{\gamma_{a}}{\psi_{a}}+\frac{\gamma_{a}}{\tilde{\psi}_{a}}-\frac{\gamma_{a}}{\rho_{a}}-1\right)\right] where we have introduced new variables γ a = ⟨ k a ​ ( x , x ) ⟩ 2 ⟨ k a ​ ( x , y ) 2 ⟩ \gamma_{a}=\frac{\left\langle k_{a}(x,x)\right\rangle^{2}}{\left\langle k_{a}(x,y)^{2}\right\rangle} , ψ a ≔ ⟨ k a ​ ( x , x ) ⟩ 2 ⟨ k a ​ ( x , x ) 2 ⟩ \psi_{a}\coloneqq\frac{\left\langle k_{a}(x,x)\right\rangle^{2}}{\left\langle k_{a}(x,x)^{2}\right\rangle} , ψ ~ a ≔ ⟨ k ~ a ​ ( w , w ) ⟩ 2 ⟨ k ~ a ​ ( w , w ) 2 ⟩ \tilde{\psi}_{a}\coloneqq\frac{\left\langle\tilde{k}_{a}(w,w)\right\rangle^{2}}{\left\langle\tilde{k}_{a}(w,w)^{2}\right\rangle} , and ρ a = ⟨ ϕ a ​ ( x , w ) 2 ⟩ 2 ⟨ ϕ a ​ ( x , w ) 4 ⟩ \rho_{a}=\frac{\left\langle\phi_{a}(x,w)^{2}\right\rangle^{2}}{\left\langle\phi_{a}(x,w)^{4}\right\rangle} . Each of them is participation ratio (PR), i.e. effective/soft count, of some quantities. For discrete quantities, let us define PR as ( ∑ i a i ) 2 ∑ i a i 2 . \frac{\left(\sum_{i}a_{i}\right)^{2}}{\sum_{i}a_{i}^{2}}. It is easy to see that, if N N number of a i a_{i} ’s take value 1 1 and the rest 0 0 , then PR is N N , indicating it is a soft count of non-zero a i a_{i} . The continuous version is