Sparse Autoencoders Find Highly Interpretable Features in Language Models — part 3/3

Cleaned-text code points [31456, 40916). Bibliography omitted here; source.md includes it.

A possible concern is that the autointerpretability method described in Section 3 combines top activating fragments (which are usually large) with random activations (which are usually small), making it relatively easy to identify activations. Following the lead of Bills et al. (2023) , we control for this by recomputing the autointerpretation scores by modifying Step 3 using only randomly selected fragments. With large sample sizes, using random fragments should be the true test of our ability to interpret a potential feature. However, the features we are considering are heavy-tailed, so with limited sample sizes, we should expect random samples to underestimate the true correlation.

In Figure 9 we show autointerpretability scores for fragments using only random fragments. Matching Bills et al. (2023) , we find that random-only scores are significantly smaller than top-and-random scores, but also that our learned features still consistently outperform the baselines, especially in the early layers. Since our learned features are more sparse than the baselines and thus, activate less on a given fragment, this is likely to underestimate the performance of sparse coding relative to baselines.

An additional potential concern is that the structure of the autoencoders allows them to be sensitive to less than a full direction in the activation space, resulting in an unfair comparison. We show in Appendix G that this is not the source of the improved performance of sparse coding.

While the residual stream can usually be treated as a vector space with no privileged basis (a basis in which we would expect changes to be unusually meaningful, such as the standard basis after a non-linearity in an MLP), it has been noted that there is a tendency for transformers to store information in the residual stream basis ( Dettmers et al., 2022 ) , which is believed to be caused by the Adam optimiser saving gradients with finite precision in the residual basis ( Elhage et al., 2023 ) . We do not find residual stream basis directions to be any more interpretable than random directions.

### C.3 Interpreting the MLP Sublayer

Our approach of learning a feature dictionary and interpreting the resulting features can, in principle, be applied to any set of internal activations of a language model, not just the residual stream. Applying our approach to the MLP sublayer of a transformer resulted in mixed success. Our approach still finds many features that are more interpretable than the neurons. However, our architecture also learns many dead features, which never activate across the entire corpus. In some cases, there are so many dead features that the set of living features does not form an overcomplete basis. For example, in a dictionary with twice as many features as neurons, less than half might be active enough to perform automatic interpretability. The exceptions to this are the early layers, where a large fraction of them are active.

For learning features in MLP layers, we find that we retain a larger number of features if we use a different matrix for the encoder and decoder, so that Equations 1 and 2 become

𝐜 \displaystyle\mathbf{c} = \displaystyle= R ​ e ​ L ​ U ​ ( M e ​ 𝐱 + 𝐛 ) \displaystyle ReLU(M_{e}\mathbf{x}+\mathbf{b}) (5) 𝐱 ^ \displaystyle\mathbf{\hat{x}} = \displaystyle= M d T ​ 𝐜 \displaystyle M_{d}^{T}\mathbf{c} (6)

We are currently working on methods to overcome this and find truly overcomplete bases in the middle and later MLP layers.

### C.4 Interpretability Scores Correlate with Kurtosis and Skew of Activation

It has been shown that the search for sparse, overcomplete dictionaries can be reformulated in terms of the search for directions that maximise the ℓ 4 \ell^{4} -norm ( Qu et al., 2019 ) .

We offer a test of the utility of this by analysing the correlation between interpretability and a number of properties of learned directions. We find that there is a correlation of 0.19 and 0.24 between the degree of positive skew and kurtosis respectively that feature activations have and their top-and-random interpretability scores, as shown in Table 2 .

This also accords with the intuitive explanation that the degree of interference due to other active features will be roughly normally distributed by the central limit theorem. If this is the case, then features will be notable for their heavy-tailedness.

This also explains why Independent Component Analysis (ICA), which maximises the non-Gaussianity of the found components, is the best performing of the alternatives that we considered.

## Appendix D Qualitative Feature Analysis

### D.1 Residual Stream Basis

Figure 11 gives a token activation histogram of the residual stream basis. Connecting this residual stream dimension to the apostrophe feature from Figure 4 , this residual dimension was the 10th highest dimension read from the residual stream by our feature 7 7 7 The first 9 did not have apostrophes in their top-activations like dimension 21. .

### D.2 Examples of Learned Features

Other features are shown in Figures 12 , 13 , 14 , and 15 .

### D.3 Feature Search Details

We searched for the apostrophe feature using the sentence “ I don’t know about that. It is now up to Dave”’, and seeing which feature (or residual stream dimension) activates the most for the last apostrophe token. The top activating feature in our dictionary was an outlier dimension feature (i.e., a feature direction that mainly reads from an outlier dimension of the residual stream), the apostrophes after O (and predicted O’Brien, O’Donnell, O’Connor, O’clock, etc), then the apostrophe-preceding-s feature.

For the residual basis dimension, we searched for max and min activating dimensions (since the residual stream can be both positive and negative), where the top two most positive dimensions were outlier dimensions, the top two negative dimensions were our displayed one and another outlier dimension, respectively.

### D.4 Failed Interpretability Methods

We attempted a weight-based method going from the dictionary in layer 4 to the dictionary in layer 5 by multiplying a feature by the MLP and checking the cosine similarity with features in layer 5. There were no meaningful connections. Additionally, it’s unclear how to apply this to the Attention sublayer since we’d need to see which position dimension the feature is in. We expected this failed by going out of distribution.

## Appendix E Number of Active Features

In Figure 16 we see that, for residual streams, we consistently learn dictionaries that are at least 4x overcomplete before some features start to drop out completely, with the correct hyperparameters. For MLP layers you see large numbers of dead features even with hyperparameter α = 0 \alpha=0 . These figures informed the selection of α = 8.6 ​ e − 4 \alpha=8.6e-4 and α = 3.2 ​ e − 4 \alpha=3.2e-4 that went into the graphs in Section 3 for the residual stream and MLP respectively. Due to the large part of the input space that is never used due to the non-linearity, it is much easier for MLP dictionary features to become stuck at a position where they hardly ever activate. In future we plan to reinitialise such ‘dead features’ to ensure that we learn as many useful dictionary features as possible.

## Appendix F Editing IOI Behaviour on other Layers

In Figure 17 we show results of the procedure in Section 4 across a range of layers in Pythia-410M.

## Appendix G Top K Comparisons

As mentioned in Section 3 , the comparison directions learnt by sparse coding and those in the baselines are not perfectly even. This is because, for example, a PCA direction is active to an entire half-space on one side of a hyperplane through the origin, whereas a sparse coding feature activates on less than a full direction, being only on the far side of a hyperplane that does not intersect the origin. This is due to the bias applied before the activation, which is, in practice, always negative. To test whether this difference is responsible for the higher scores, we run a variant of PCA and ICA in which we have a fixed number of directions, K, which can be active for any single datapoint. We set this K to be equal to the average number of active features for a sparse coding dictionary with ratio R = 1 R=1 and α = 8.6 ​ e − 4 \alpha=8.6e-4 trained on the layer in question. We compare the results in Figure 18 , showing that this change does not explain more than a small fraction of the improvement in scores.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .