Rotation in Language Models: Simple Structure, Changes of Basis, and Conceptors

1Introduction

A transformer language model stores each token of its vocabulary as a row of numbers, and processes text by repeatedly rewriting a table of such rows, one per token position, through a stack of layers. Every row lives in the same vector space, called the residual stream, and the coordinates of that space are arbitrary in one precise sense: a rotated copy of the model, with the weights rotated to match, computes exactly the same function. Any description of the stream therefore begins, whether it says so or not, by choosing coordinates. This is the first thing a psychometrician recognizes. A loading matrix is likewise defined only up to rotation, and a century of factor-analytic practice has been devoted to choosing the rotation that makes the loadings readable.

The question of this report is what happens when that practice (varimax, promax, bifactor rotation, the pattern/structure distinction, the general factor) is applied to a language model, at each of the four places it can be applied: to the table of token vectors the model starts from (Section 3); to the model as a whole, which raises the question of whether an oblique transformation, not a rotation but an arbitrary invertible change of coordinates, can be used without changing it (Section 4); to a conceptor (Jaeger, 2014), a soft projection matrix fitted to a layer's activations and used to read and steer what the model is doing (Section 5); and to a layer, the activations themselves (Section 6). The conceptor is the application that motivates the work, and it is also what ties the four places together, because a conceptor is a factor-analytic object: it is built from the covariance matrix of a layer's activations, and its eigen-decomposition is an unrotated principal-components solution, so everything that factor analysis knows about rotating such a solution applies to it. Section 7 turns the findings into proposals for conceptor practice.

The tone is deliberately didactic. Derivations are carried out in full, assumptions are stated, and every numerical claim comes from a computation on GPT-2 small (Radford et al., 2019) that can be re-run from the accompanying scripts given the public model weights (Appendix A). The intended reader knows factor analysis and is willing to learn the two or three facts about transformers that matter here; Section 2 supplies them. Readers who know transformers but not factor analysis will find the rotation facts they need in the same section.

1.1 What is established, and what this report adds

Rotating a word-embedding table to interpretable axes is not new. Park, Bak and Oh (2017) applied the Crawford–Ferguson family, in orthogonal and oblique form, to full skip-gram and GloVe tables and scored the axes by an automated word-intrusion test; Hollis and Westbury (2016) and Westbury, Yang and Anderson (2025) ran principal components and factor analysis on word2vec from the psychology side; the ICA line (Yamagiwa, Oyama & Shimodaira, 2023; Musil & Mareček, 2024) finds interpretable axes by a different rotation criterion; Rothe, Ebert and Schütze (2016) and Dufter and Schütze (2019) learn orthogonal transformations that concentrate a labelled property on a few axes; and Rohe and Zeng (2023) supplied the statistical theory under which principal components followed by varimax recovers sparse latent factors. Sparse overcomplete codings of a table (Murphy, Talukdar & Mitchell, 2012; Faruqui et al., 2015; Subramanian et al., 2018) and sparse autoencoders for a layer's activations (Bricken et al., 2023; Huben et al., 2024) are the overcomplete, nonlinear descendants of the same idea, and Arora et al. (2018) model a word vector as a sparse sum of "discourse atoms", which is the cross-loading phenomenon of Section 3 seen from a generative model. That embedding spaces have a dominant common direction, and why, is also known: Gao et al. (2019) showed that likelihood training with tied embeddings drives the vocabulary into a narrow cone, and Biś, Podkorytov and Liu (2021) traced the drift to the gradient that every non-target token receives at every step; the outlier coordinates this produces are documented by Kovaleva et al. (2021), Timkey and van Schijndel (2021) and Puccetti et al. (2022), and the anisotropy of contextual representations by Ethayarajh (2019) and Mu and Viswanath (2018). Conceptors have been applied to embedding tables, notably by Liu, Ungar and Sedoc (2019a), whose "conceptor negation" removes a table's dominant directions, and to debiasing (Karve, Ungar & Sedoc, 2019; Yifei, Ungar & Sedoc, 2023). Section 3 is, in that light, a replication on a transformer's table with the full psychometric toolkit; its value is in the detail, and in modelling the general factor as one.

That a transformer is unchanged by an orthogonal rotation of its residual stream is also established. Elhage et al. (2021) note the freedom to reparameterize the stream and the related freedom inside each attention head (only the products W_QW_K^T and W_OW_V are determined); Elhage, Lasenby and Olah (2023) state the residual-stream version as the absence of a privileged basis; SliceGPT (Ashkboos, Croci et al., 2024) proves it as a theorem, handles LayerNorm by converting it to RMSNorm, and uses it to compress models; and QuaRot (Ashkboos, Mohtashami et al., 2024) uses the same invariance with Hadamard rotations to remove outliers before quantization. Three things need keeping apart here: that a model admits equivalent coordinate descriptions (exact); that the coordinates it was trained in nevertheless carry empirical structure, which Elhage et al. (2023) document and Section 3.5 meets in the form of outlier coordinates; and that a different basis may be more useful for description. The first does not deny the second, and the third is what this report is about.

Against that background, what this report adds is modest in algebra and, we think, useful in practice:

1. The oblique change of basis, stated and verified (Section 4). Extending the orthogonal invariance to any invertible transformation is a small step once the normalization layer is given the induced metric; what the step buys is a reading, that the input and output tables then stand in the pattern-to-structure relation of oblique factor analysis, and a warning, that the rewrite which forgets the metric is a different model, which the verification on GPT-2 makes concrete.
2. The transformation law of conceptors (Section 5). Under a change of basis a conceptor transforms as M⁻¹CM; its gains, quota, action and Boolean-style operations are invariant; the aperture regularizer is not, and the fitting formula in oblique coordinates acquires the metric. The algebra is elementary; the consequence, that a conceptor fitted naively in oblique coordinates is wrong by tens of percent while looking reasonable, is the practical point.
3. Conceptors read on rotated axes (Section 6). A conceptor's gain along each axis of a rotated, interpretable basis of its layer is a labelled directional summary of what it keeps, and rotation inside the conceptor's own subspace yields an unsupervised axis that separates held-out sentences by the concept, though not as sharply as the supervised direction itself, and names the nuisance variation beside it. What rotation adds there is names, not separation, and the report says so.
4. Where in the model a concept is, measured against baselines (Section 6). Against random-direction chance, a sentiment concept is absent from the leading directions of a general corpus until the middle of the model; but a label-shuffled null rises with depth in the same way, and against a reference set matched to the concept's own kind of state the concept stands out at one layer only, the one at which the sentences are also best separated. The baselines, not the depth trend, are what bears on choosing a layer for conceptor fitting.

Section 7 turns these into proposals for conceptor practice. They are proposals; none has been tested beyond the demonstrations here.

2Background: the residual stream, factor rotation, conceptors

This section fixes notation and collects the facts used later. Vectors are rows throughout, so that a linear map acts by right-multiplication, h → hM; this is the convention of the model's own code and it keeps the factor-analytic formulas in their familiar shape.

2.1 The residual stream of a transformer

GPT-2 small has a vocabulary of V = 50,257 tokens and a width of d = 768. Its lookup table E is a V × d matrix, one row per token; a second table P holds one row per position. Reading a text of T tokens, the model forms the T × d matrix H₀ whose t-th row is E_token(t) + P_t, and then applies twelve blocks, each of which adds an update to every row:

The sequence of H matrices, from H₀ to H₁₂, is the residual stream; "layer ℓ" below means the state after ℓ blocks, so layer 12 is the state after the last block and before the final normalization. The attention and MLP sub-blocks each begin by reading the stream through weight matrices (the query, key and value maps W_Q, W_K, W_V, and the MLP's input map W_in, all of shape d × something) and end by writing to it through others (W_O and W_out, of shape something × d). Everything in between, the softmax over attention scores and the GELU nonlinearity, acts on the read-out values, never on the stream's coordinates directly. The one exception is the normalization layer. GPT-2 uses LayerNorm (Ba, Kiros & Hinton, 2016):

where 1 is the all-ones row vector, γ and β are learned elementwise gains and biases, and ε = 10⁻⁵ is a stabilizer. LayerNorm therefore does two things that depend on the coordinate system: it subtracts the mean along the all-ones direction, and it divides by a length measured with the Euclidean metric. Modern models (Llama, Qwen, Gemma) use RMSNorm (Zhang & Sennrich, 2019), which omits the centring; the difference matters in Section 4. Finally the output is read through the tied unembedding: logits = LN_f(H₁₂) E^T, the same table used backwards.

Nothing above names a coordinate. If every row of the stream is replaced by hQ for an orthogonal Q, and every weight that reads is replaced by Q^TW, every weight that writes by WQ, the reads and writes are unchanged and so is the model; the only thing that could notice is LayerNorm, through the all-ones direction. Elhage, Lasenby and Olah (2023) call this the absence of a privileged basis, while showing that the coordinates a model is trained in do acquire empirical structure (a few of GPT-2's coordinates carry unusually large activations, a point Section 3.5 returns to); SliceGPT (Ashkboos, Croci et al., 2024) states the invariance as a theorem and uses it. The consequence for this report is modest and exact: a rotation chosen for interpretability is as legitimate a description of the model as the trained coordinates, in the way a rotated factor solution is as legitimate as the unrotated one.

2.2 Factor rotation in one page

Let Λ be a p × k loading matrix: p indicators, k factors. Any orthogonal k × k matrix Q gives an equally good solution ΛQ; rotation chooses Q to make the loadings readable, usually by Thurstone's (1947) simple structure: each indicator loads on few factors and each factor on few indicators. Varimax (Kaiser, 1958) maximizes, over orthogonal Q, the sum over factors of the variance of the squared loadings; with Kaiser normalization each row of Λ is first scaled to unit length so that long rows do not dominate. Oblique rotations allow correlated factors. Writing the transformation as an invertible T, the pattern matrix is Λ_p = ΛT (regression weights of indicators on factors), the factor correlation matrix is Φ = (T^TT)⁻¹, and the structure matrix is Λ_s = Λ_pΦ = ΛT^−T (correlations of indicators with factors). For orthogonal T the two coincide. The oblique method used here is promax (Hendrickson & White, 1964): raise the varimax loadings to a power (4) to sharpen them, fit that target by least squares, and rescale. (Direct oblimin, Jennrich & Sampson, 1966, was also run on the full table; it recovered the same factors with the same conclusions, and is omitted to keep the comparison to one oblique method.)

When a solution has a general factor, a direction almost every indicator shares, neither orthogonal nor oblique simple-structure rotation represents it well: an orthogonal rotation must put it in one column, which then dominates, and an oblique rotation disperses it into the factor correlations. The model built for this situation is the bifactor model (Holzinger & Swineford, 1937; Reise, Moore & Haviland, 2010): one general factor on which every indicator loads, plus group factors each indicator loads on at most one of, all mutually orthogonal. The exploratory version used here is the bi-quartimin rotation of Jennrich and Bentler (2011): rotate Λ orthogonally to minimize the quartimin criterion applied to all columns but the first, Σ_i Σ_1<r<s λ_ir²λ_is², which is zero exactly when the group columns have perfect cluster structure and leaves the first column free to hold whatever is common. It is computed by the gradient projection algorithm (Jennrich, 2001; Bernaards & Jennrich, 2005). Jennrich and Bentler show that the older Schmid–Leiman (1957) procedure, which first fits an oblique solution and then orthogonalizes it through a second-order factor, imposes proportionality constraints and can fail badly when the data have genuine bifactor structure (see also Reise, Moore & Haviland, 2010; Abad et al., 2017); it is not used here. As a check on the analytic rotation, which is known to be prone to local minima and to shift variance onto the general factor (Mansolf & Reise, 2016), a partially specified target rotation (Browne, 2001) was also run, with the general column unspecified and the group columns given zeros wherever the varimax loading fell below 0.2.

Simple structure is measured in three ways below: Hofmann's (1978) row complexity (Σλ²)²/Σλ⁴, which is 1 for an indicator loading on one factor and k for one loading equally on all; the number of salient loadings per row (|λ| ≥ 0.3); and the share of a row's squared coefficients carried by its largest coefficient. For an orthogonal solution of unit-length rows the last is the share of the row's squared length on one axis; for oblique pattern coefficients, whose squares do not sum to a length, it is a coefficient-concentration index and is labelled as such.

One more distinction is needed, because the matrices here are tall and nobody designed their columns. In R-mode analysis the columns are variables and the rows observations; factors are directions in variable space. In Q-mode analysis (Stephenson, 1935; Cattell, 1952) the roles are reversed: the rows are variables, the columns observations, and factors are clusters of rows. For a token table the rows are tokens and the columns are the model's 768 arbitrary coordinates, and Section 3 shows that only Q-mode produces anything readable.

2.3 Conceptors in one page

A conceptor (Jaeger, 2014, 2017) is fitted to a set of N activation rows X (N × d), here the residual-stream rows of example sentences at one layer. With c the mean row and R = (X − c)^T(X − c)/N the covariance matrix of the examples (Jaeger calls it the correlation matrix; the coordinates are not standardized),

where R = U diag(s_j)U^T is the eigen-decomposition. The gains s_j/(s_j + λ) lie in [0, 1); directions along which the examples vary a lot are kept, directions they do not use are muted. The aperture α = λ^−1/2 sets how picky the filter is; this report uses λ = tr(R)/d. The quota tr(C)/d is the mean gain, which Triantafyllopoulos et al. (2026) use as an unsupervised diagnostic for choosing a layer. A conceptor is applied by multiplication, and the published steering rules differ in what is multiplied: Postmus and Abreu (2024) replace the activation by βhC; Triantafyllopoulos et al. (2026) use replacement or the interpolation (1 − β)h + βhC; Moore (2026), whose sentences and conceptors are reused here, centres and amplifies, h + t(h − c)C, and filters by c + (h − c)C. This report uses the last two. Conceptors can be combined by Boolean-style operations: NOT C = I − C; C₁ AND C₂ = (C₁⁻¹ + C₂⁻¹ − I)⁻¹ when the inverses exist, with Jaeger's (2014) general definition through pseudo-inverses otherwise; OR by De Morgan. These operations are not a Boolean algebra in the strict sense (AND is not idempotent, for instance), which is why "Boolean-style" is used here. The conceptors in this report are fitted to 96 or 16 centred rows in 768 dimensions and so have rank at most 95 or 15; wherever AND is computed, a ridge of 10⁻⁸I is added before inversion, and all Boolean results below are for that regularized operation. Equation (3) is an unrotated principal-components solution: U is the principal-component basis of the examples, and the gains are a monotone function of the eigenvalues. Everything factor analysis says about rotating such a solution therefore has a conceptor reading, which is the subject of Section 5.

2.4 Why simple structure has a ceiling: superposition

Any single vector can be made one-factorial by choosing a basis with an axis along it. The difficulty is a common basis for 50,257 vectors at once. Elhage et al. (2022) showed that a network pressed to represent more features than it has dimensions stores them as nearly, but not exactly, orthogonal directions, so that a typical activation is a dense superposition of several features; and if each token's row is a sum of several features that are shared across tokens, then a basis that makes one token sparse will not make its neighbours sparse, and the best any rotation can do for the collection is to align axes with the features and accept that a token loads on as many axes as it has features. The token table is a case in point: legends is at once a plural noun and a word about myth, and GPT-2 writes both into one 768-vector. This sets expectations for Section 3: rotation will sharpen the table a great deal, and cross-loadings will remain. Whether a given cross-loading reflects a shared feature, a tokenization artefact, or imperfect optimization is not something the rotation itself can say; the examples in Section 3 are read as shared features because they are legible as such, which is an interpretation.

3Rotating the lookup table

The lookup table E of GPT-2 small is a 50,257 × 768 matrix of real numbers; its rows have lengths between 2.5 and 6.3, and two tokens chosen at random have a mean cosine similarity of 0.27, the anisotropy first documented for contextual models by Ethayarajh (2019). This section treats E as a loading matrix and asks what rotation does to it.

3.1 Three ways to factor a token table, and which one to rotate

There are three natural analyses of a V × d table.

• R-mode: the 768 columns are variables, standardized across tokens; the tokens are observations. The correlation matrix is 768 × 768, extraction gives directions in the model's coordinate space, and the pattern matrix is 768 × k: how each coordinate loads on each factor.
• Q-mode: the tokens are variables, each standardized across its 768 coordinates; the coordinates are observations. The correlation between two tokens is then their cosine similarity after row-centring, so the analysis factors the token-similarity matrix; the pattern matrix is V × k: how each token loads on each factor.
• Full rotation: take E itself as the loading matrix (tokens = indicators, k = d = 768 factors), scale its rows to unit length, and rotate. This is Q-mode without the reduction to k < d factors and with unit rows in place of standardized ones.

The three use different preprocessing (column standardization, row standardization, unit rows) and so are not literally one singular value decomposition seen from three sides, but their leading subspaces nearly coincide, and the point of listing them is which side receives the simple-structure criterion. In R-mode, rotation asks "which factor does coordinate 447 belong to?", a question with no answer because the coordinates are arbitrary (Section 2.1); the criterion is spent on noise. In Q-mode it asks "which cluster does the token youth belong to?", which is the question one wants answered. The empirical confirmation is in Section 3.5: an R-mode pattern matrix of GPT-2 is a wall of coefficients near 0.1, interpretable only through its first factor, which is a handful of outlier coordinates. Everything else in this section is Q-mode or the full rotation, and the two agree: the full rotation is the finest resolution (one factor per word family), the k = 30 Q-mode solution the coarsest (one factor per broad category).

3.2 Varimax on the full table

Rows were scaled to unit length (Kaiser normalization) and the 50,257 × 768 matrix was rotated by varimax for 600 iterations of the singular-value algorithm (42 minutes on two CPU cores; the criterion rose by 0.03% over the last thirty iterations, so the solution is stable rather than formally converged). Figure 1(a) shows what happened to six tokens: in the trained coordinates their largest coordinate carries 2–3% of their squared length; after rotation one coordinate carries 30–50%. Over all tokens the mean share of squared length in the largest coordinate rises from 0.022 to 0.166, and 58% of tokens have exactly one loading above 0.3 (49% of ordinary English words, defined throughout as lower-case alphabetic tokens with a leading space and a frequency in the wordfreq lexicon, n = 18,770). Table 1 shows what the factors are. A factor is, most often, a word family: by a crude rule (the five highest-loading tokens share their first three letters after stripping the leading space) 343 of the 768 are, and the examples show the rule under-counts, since youth, youngsters, teenagers, young, teen, kids, children is a family the rule misses. The orthographic variants of a word (capitalized, with and without a leading space, plural) cluster together, which says that the table stores "which word" strongly and its surface form as a small offset. The remainder are format factors (three-digit numbers, years, HTML brackets, ALL-CAPS words, sentence-initial capitalized words, two-letter fragments), grammatical factors (past-tense verbs, -ing forms, prepositions), and semantic factors broader than a family (European nationalities; medical vocabulary; superlative adjectives).

Cross-loadings remain, as Section 2.4 predicted, and the legible ones pair a form with a topic: legends loads on the plural-noun factor and on the mythology factor; failed on past tense and on collapse; percentages on plurals and on statistical vocabulary. Roughly half the factors describe form (number, tense, capitalization, format) and half describe topic, and a typical word has one loading of each kind. To the extent this holds, the table is close to additive in form and meaning, which is why one-factor structure is unreachable.

3.3 Promax does not reduce cross-loadings, and its factor correlations hide a general factor

Promax was applied to the varimax solution with the usual power of 4. Table 2 gives the simple-structure indices. On every measure varimax is the sharper: it puts the larger share of each token's coefficients on one factor, and 58% of tokens have exactly one salient loading against 32% for promax, which lets 57% of tokens cross-load. The two recover the same factors where they were compared (the youth family is varimax F96 and promax F29 with the same ten tokens at the top; likewise dead, the -ify suffix, both/neither, years, medical, negative affect), so the difference is in the bookkeeping of cross-loadings, not in what the axes mean; the comparison was made on these examples, not on all 768 axes.

The factor correlation matrix is the diagnostic that decides what to do next. Pairwise, the promax factors barely correlate (mean |φ| = 0.065), which agrees with Park et al. (2017), who found orthogonal rotations better than oblique ones for interpretability of word-vector axes, and with what superposition predicts: when every row is a sum of several nearly orthogonal features, the sparsest coordinate system is itself nearly orthogonal. But a correlation matrix can be close to the identity pairwise and far from it collectively, and this one is: its largest eigenvalue is 28.4 against 18.6 for the next and a smallest near 10⁻⁶ (promax split a tight core of some sixty unusual byte tokens into two near-collinear factors with r = −0.98, an ill-conditioned solution rather than a Heywood case). A dominant eigenvalue of Φ is what a general factor looks like after an oblique rotation has dispersed it into the factor correlations. That is the cue to stop comparing simple-structure rotations and fit the model built for a general factor.

3.4 The general factor: varimax, promax, and the bifactor rotation

Embedding spaces have a dominant common direction, and the mechanism is understood: under likelihood training with a tied output embedding, every token that is not the target at a given step receives a gradient in the same direction, and the accumulated drift pushes the vocabulary, the rarely-targeted tokens most of all, into a narrow cone (Gao et al., 2019; Biś, Podkorytov & Liu, 2021; see also Mu & Viswanath, 2018). In the Q-mode analysis of this table the first eigenvalue of the token-similarity matrix is 13,625 against 443 for the second: 27% of the variance in one component, 0.9% in the next (Figure 2(a)). A spike of that size is a near-rank-one component that nearly every token shares with the same sign, and it behaves exactly as a general factor behaves in psychometrics. Three rotations treat it three ways.

Varimax, being orthogonal, must put the common direction somewhere, and puts it in one factor: in the full solution the largest factor is the primary factor for 71% of all tokens and 65% of ordinary words, and in the k = 30 Q-mode solution one factor is primary for 82% of tokens. Promax partly disperses it (20% of tokens remain on its largest factor) but leaves it visible in two places at once: in the 30-factor Q-mode promax solution, factor 1 is primary for 18,573 tokens and correlates 0.5–0.66 with the next six (Figure 2(b)). A power-4 target built from a varimax start cannot parcel out a direction that almost every row genuinely contains, which is the situation Jennrich and Bentler (2011) describe. Bi-quartimin gives the general factor a column of its own. On the 30 leading Q-mode components the rotation converged to the same criterion from the principal-components start and from ten random orthogonal starts (criterion 0.01079 to 0.01083, a spread of 0.4%), so the local-minimum concern of Mansolf and Reise (2016) did not materialize here; the target rotation described in Section 2.2 recovered the same general factor (r = 0.99 between the two general columns) and group factors that match the bi-quartimin ones at a mean absolute column correlation of 0.75. Table 3 summarizes the solution.

The bifactor general factor loads positively on every one of the 50,257 tokens, with a mean of 0.52 and a narrow spread (0.43 to 0.60 between the 5th and 95th percentiles); the full-table version has the same mean and the same ordering of tokens (range 0.14 to 0.90). At the top are byte-level and non-Latin fragments and code-like fragments (externalToEVA, quickShip); at the bottom are the most frequent function words (the 0.14, and 0.16, a 0.18, in 0.19). Over 18,770 ordinary words the general loading correlates −0.55 with Zipf frequency (Figure 2(c)); the relation is flat below Zipf 4 and steep above it, so it is the frequent words that have moved away from the common direction rather than the rare ones that have moved toward it, which is what the gradient account of Gao et al. (2019) and Biś et al. (2021) predicts: frequently targeted tokens are the ones whose embeddings are repeatedly pulled off the common drift. The correlation is with an external frequency estimate, not with the model's training frequencies, which are unknown, and the tight cluster of byte tokens at the top is better described as a set of unusual tokens whose embeddings sit close together than as untrained, since even a token never seen as input receives gradient through the tied unembedding. It is the same structure Puccetti et al. (2022) and Timkey and van Schijndel (2021) describe from the dimension side, and it connects to conceptors: Liu, Ungar and Sedoc's (2019a) conceptor negation fits a conceptor to the whole table and applies NOT, which softly removes exactly these dominant directions, the "all-but-the-top" post-processing of Mu and Viswanath (2018) done with a conceptor.

The group factors are the second dividend, and the claim for them is descriptive rather than metric: with the common direction removed, the 29 bi-quartimin group factors are the promax factors with their poles unmixed. Function words and punctuation, which under promax were the negative pole of the general factor, become a group factor of their own; the rest are word-internal fragments, administrative vocabulary, three-digit numbers, plural nouns, evaluative intensifiers, sentence-initial words, male first names, capitalized game and weapon nouns, violence and crime, past-tense verbs, Asian nationalities, chemistry and biomedicine, gender and sexuality, entertainment events, finance, ALL-CAPS, computing, the divine, -ing forms, geography, quantitative decline, research, soldiers and warriors, programming identifiers, innovation, political rhetoric, anxiety and distress, and years (Appendix B lists all 29 with their top tokens). Group loadings are modest (0.25 to 0.69 at the top of each) because the general factor has taken 71% of the common variance, and only 4.8% of tokens have a group loading above 0.3; the factors are read from their highest-loading tokens, which are unambiguous, rather than from salience counts, and no word-intrusion or other quantitative interpretability test was run on them. On the full table the picture is the same at word-family resolution: the general factor keeps 27% of the tokens' squared length and no group factor is primary for more than 2.5% of tokens, against 71% for varimax's giant factor.

3.5 The dimension side

For completeness, the R-mode analysis (768 coordinates as variables, promax, k = 30) was also run. Its pattern matrix is unreadable: the median coordinate's largest loading is 0.18 and only 7% of coordinates load above 0.4 on anything. The one exception is factor 1, which is carried by about 35 coordinates including the known outlier dimensions of GPT-2 (coordinates 138 and 447 in the usual zero-based numbering; Timkey & van Schijndel, 2021; Kovaleva et al., 2021) and which, read through its token scores, is the common direction again. Rotating the dimension side spends the simple-structure criterion on a question that has no answer; the coordinates are unprivileged in the sense of Section 2.1, and the factor analysis knows it. That the common direction nevertheless concentrates on a few trained coordinates is the empirical privilege Elhage et al. (2023) describe; the two statements are compatible.

4A change of basis leaves the model unchanged

Section 3 chose new coordinates for the lookup table. This section shows that the whole model can be carried into those coordinates, orthogonal or oblique, without changing what it computes, and says exactly what has to be done to the weights. The orthogonal case is established: Elhage et al. (2021) note the freedom, SliceGPT (Ashkboos, Croci et al., 2024) proves it for any orthogonal Q after converting LayerNorm to RMSNorm by absorbing the centring into the write matrices, and QuaRot (Ashkboos, Mohtashami et al., 2024) applies it with Hadamard rotations. Inside an attention head the freedom is larger still, since only the products W_QW_K^T and W_OW_V are determined (Elhage et al., 2021), and reparameterization symmetries of network weights are a subject of their own. The oblique case, which is what a promax solution calls for, is the residual-stream symmetry written out with LayerNorm left in place. It needs two things: the normalization layer must carry the induced metric, and the tied unembedding must be transformed by the inverse-transpose. Neither is deep. What they buy is a reading, that the input and output tables then stand in the pattern-to-structure relation of oblique factor analysis (in plain linear algebra, coordinates in a basis and inner products with it, the primal and dual descriptions of one vector), and a warning, that a rewrite which forgets the metric is a different model.

4.1 The transformation rules

Let M be any invertible d × d matrix and write the residual stream in new coordinates, h′ = hM. A read, hW, is preserved by replacing W with M⁻¹W, since hMM⁻¹W = hW. A write of a vector u into the stream is preserved by writing uM instead, so a write matrix W becomes WM and a bias b becomes bM. The lookup and position tables are writes: E′ = EM, P′ = PM. Everything between a read and a write (attention scores, softmax, GELU) sees the same numbers as before and is untouched. That leaves LayerNorm and the unembedding.

LayerNorm. Equation (2) centres h along 1 and divides by a Euclidean length. In the new coordinates the all-ones direction is 1′ = 1M. The Euclidean length of a vector v whose new coordinates are v′ = vM is ‖v‖² = v′M⁻¹M^−Tv′^T = v′Φv′^T with

which is the factor-correlation matrix of Section 2.2 when M is a rotation's pattern transform, and the identity when M is orthogonal. The mean is μ = h·1/d = h′c/d with c = M⁻¹1^T. The normalized vector in new coordinates is therefore

a LayerNorm that centres along the image of the ones vector and measures length with the metric Φ. The stabilizer ε is added to the same variance as before and needs no change. The elementwise gain γ and bias β are folded into the read that follows: LN(h)W + b = n diag(γ)W + (βW + b). The read therefore becomes M⁻¹diag(γ)W with bias b + βW, and the new LayerNorm has unit gain and zero bias.

Two special cases are worth naming. If M is orthogonal and fixes the ones direction (M^T1^T = 1^T), then Φ = I, 1′ = 1, c = 1^T, and equation (5) is ordinary LayerNorm: nothing in the architecture changes. If M is orthogonal but moves the ones direction, as a varimax rotation generally does, the metric is still the identity but the centring direction has moved, and ordinary LayerNorm is wrong. SliceGPT's route around this is to make the stream mean-free by projecting every write onto the complement of 1, after which LayerNorm is RMSNorm and any orthogonal Q is exact. Equation (5) instead keeps the stream as it is and moves the centring. For RMSNorm models there is no centring, so every orthogonal rotation is exact with the standard layer, and only the metric is needed for an oblique one.

The tied unembedding. The logits are LN_f(h)E^T = n diag(γ_f)E^T + β_fE^T. The unembedding is a read, so in the new coordinates it is U′ = M⁻¹ diag(γ_f) E^T, plus a constant logit offset β_fE^T. Set the final gain aside for a moment and compare the two tables. The input table is EM. The transposed output table is EM^−T = EM(M^TM)⁻¹ = (EM)Φ. In the language of Section 2.2, with Λ = E and T = M, the input table is the pattern matrix ΛT and the output table is the structure matrix ΛT^−T = Λ_pΦ. For an orthogonal M the two coincide. For an oblique M the model needs both faces of the same solution, pattern coordinates to read a token in and structure coordinates to score a token out, and the tie between the tables is broken. The final gain γ_f is a separate diagonal factor that sits between the stream and the unembedding in the original model as well. Folding it into the output table, as the implementation below does for convenience, makes that table the structure counterpart of E diag(γ_f) rather than of E, and breaks the literal tie even for orthogonal M; keeping it as its own layer preserves the tie exactly in the orthogonal case. This answers a natural question (can the structure matrix be used in place of the pattern matrix?) precisely: one goes in, the other comes out, and the relation between them is the factor correlation matrix.

4.2 Verification on GPT-2

The rules were implemented as a subclass of the NumPy GPT-2 used throughout this report, in double precision, and tested with three transforms: a random orthogonal rotation constructed to fix the ones direction; the varimax rotation of Section 3.2 (orthogonal, but it moves the ones direction); and the promax transform (oblique; condition number 2,439 because of the near-collinear doublet of Section 3.3, so that the induced metric Φ has condition number 6 × 10⁶, a severe test). Each was tried with ordinary LayerNorm, equation (2) applied as if the coordinates were the old ones, and with equation (5). For three prompts the logits at every position were compared with the original model's, and twenty tokens were generated greedily from each prompt.

The identity of Section 4.1 was also checked directly: the output table of the promax model equals the structure counterpart of the gain-scaled input table, (E diag(γ_f))M(M^TM)⁻¹, to 3 × 10⁻¹². Two features of Table 5 deserve emphasis. First, the naive rewrites are not slightly wrong; they are a different model, with no agreement on the most probable token at any position of the worst prompt and logit errors of order 10². The only place a transformer can notice its coordinate system is the normalization layer, and it notices emphatically. Second, the varimax row is a warning for the orthogonal case too: a rotation chosen for interpretability will in general move the all-ones direction, and a model rewritten with it needs either the centring of equation (5) or SliceGPT's mean-free conversion, unless it uses RMSNorm. The random-orthogonal row shows that the issue is the ones direction and nothing else.

4.3 What the result is good for

The proposition says that the model is indifferent to the coordinates its residual stream is written in, and Section 3 says that some coordinate systems are far more readable than the trained one. Together they license a programme: rewrite the model in a rotated basis once, and then read its weights, not only its activations, in interpretable terms. A rotated reading weight M⁻¹diag(γ)W_Q says which interpretable axes an attention head's queries attend to; a rotated writing weight W_OM says which axes it writes to; the rotated unembedding says which axes raise which tokens' logits. None of this changes the model's behaviour, so it is pure description, in the way that choosing a rotation is pure description of a factor solution. The oblique extension matters for one reason: the best descriptive basis may be oblique (Section 7 argues that it is, for correlated concepts), and the proposition says an oblique basis can be used at the price of carrying Φ through the normalization layer, which is a one-line change to the code. Sparse autoencoders (Bricken et al., 2023) pursue the same descriptive goal with an overcomplete, nonlinear dictionary that cannot be folded back into the model; a rotation can.

5Conceptors under a change of basis

A conceptor is a matrix that acts on the residual stream, so Section 4's change of basis acts on it too. This section works out how, separates what is invariant from what is not, and finds that the one non-invariant piece, the aperture regularizer, is exactly the piece a practitioner would get wrong by fitting in rotated coordinates without thinking.

5.1 The transformation law and its invariants

Let a conceptor C act on centred rows by v → vC. In new coordinates v′ = vM the same operation must send v′ to (vC)M = v′M⁻¹CM, so the transported conceptor is

a similarity transform. Everything that is preserved by similarity is therefore basis-independent: the gains (eigenvalues of C), the quota tr(C)/d, the action on any row, and the Boolean-style operations, since M⁻¹(I − C)M = I − C′, M⁻¹(C₁⁻¹ + C₂⁻¹ − I)⁻¹M = (C′₁⁻¹ + C′₂⁻¹ − I)⁻¹, and the ridge of Section 2.3 transports with them (M⁻¹(C + εI)M = C′ + εI). Overlaps between conceptors measured as traces of products are likewise invariant, and so is the quota-by-layer diagnostic of Triantafyllopoulos et al. (2026). What changes is the representation: the eigenvectors move with the basis, and for an oblique M the transported matrix is no longer symmetric (its largest asymmetry in the example below is 12), though it is self-adjoint with respect to the metric, C′Φ = (C′Φ)^T (Appendix C.4; verified to 4 × 10⁻¹²). For an orthogonal Q, C′ = Q^TCQ is symmetric and everything is as before.

5.2 Fitting in rotated coordinates: the aperture carries a metric

The subtlety is in equation (3). Written in new coordinates, the covariance matrix of the examples is R′ = M^TRM, and the obvious thing to do, R′(R′ + λI)⁻¹, is not C′. The term λI is a ridge penalty that is isotropic in the old coordinates, "the same unit length in every direction"; the identity operator is still the identity under similarity, but an isotropic penalty in the old metric is represented in the new coordinates by λM^TM = λΦ⁻¹. Substituting R = M^−TR′M⁻¹ into (3) and (6) gives the correct fitting formula in oblique coordinates,

which reduces to (3) when Φ = I. The default rule for λ must be transported too, because tr(R) = tr(ΦR′) ≠ tr(R′); for the promax transform below, tr(R′)/d would give λ = 943 where the original is 1.385, and tr(ΦR′)/d gives 1.385 exactly.

Table 6 and Figure 4 test this on the sentiment conceptor of Moore (2026), fitted at layer 8 to the final-token rows of 48 positive and 48 negative sentences (λ = 1.385, quota 0.076), transported by the promax transform of Section 3 (Φ far from the identity, condition number 6 × 10⁶). The outputs of the filters are compared on 24 evaluation rows: the 16 neutral sentences, which are outside the fitting set, and 8 of the fitting sentences. The transported operator (6) reproduces the original filter's output to 3 × 10⁻¹⁴; the metric formula (7) reproduces it to 7 × 10⁻¹⁰, the residue of inverting an ill-conditioned M. The naive refit with the same λ is a different filter: its output differs from the original by 36% of the centred output vector on average (6.5% at the worst coordinate), its quota is 0.108 rather than 0.076, and its gains, in Figure 4, are uniformly too high; the naive refit with the untransported default rule errs the other way (quota 0.013, gains collapsed, mean output error 77%). For an orthogonal rotation the naive refit coincides with the transported operator to 4 × 10⁻¹⁵, as it must. The regularized AND and NOT were checked as well: computed in the new coordinates they agree with the transported originals to 10⁻¹².

5.3 What this means for practice

Three rules follow. (i) A conceptor fitted in the trained coordinates can be carried into any rotated basis by equation (6), and nothing about what it does changes; rotation is a reading device, never a modelling decision. (ii) If a conceptor is to be fitted in rotated coordinates, as it would be if a model were stored in an interpretable basis (Section 4.3), an orthogonal basis needs no change to the fitting code and an oblique one needs equation (7) with its transported default; forgetting the metric produces a filter that is wrong by tens of percent while looking perfectly reasonable. (iii) The quantities one reports about a conceptor, its gains, its quota, its overlaps with other conceptors, are coordinate-free once the conceptor is transported correctly, so published results that report them do not depend on the basis the model was trained in; quantities computed in a basis, such as Euclidean distances between transformed rows, a naive refit, or the choice of axes to read, are not invariant, as this section's own demonstrations show. The only coordinate-dependent thing about a correctly transported conceptor is the description of what it keeps, which is exactly the thing rotation is for, and which Section 6 takes up.

6Rotating a layer

The lookup table enters the model only at the start, as one of the two terms of the initial residual state. By layer 8 a row no longer says which token it is; it says something about the sentence so far, and a conceptor fitted there is a filter on that. Rotating the table says nothing about it. But the technique transfers: collect the rows a layer produces over a corpus, and rotate those. This section does so at five depths, labels the resulting axes, and then reads the sentiment arrow and conceptor of Moore (2026) on them. The question in the background is the one every conceptor practitioner faces: which layer, and why. The answer this section reaches is less than the question hoped for, and the baselines that cut it down are the useful part.

6.1 Method

GPT-2 was run over 41,984 tokens from six public-domain books (Austen's Emma, Carroll's Alice, Chesterton's The Man Who Was Thursday, Melville's Moby-Dick, Bryant's Stories to Tell to Children, Whitman's Leaves of Grass), in windows of 128 tokens, and the residual-stream row of every token was kept at layers 2, 5, 8, 11 and 12 (layer 12 being the state after the last block, before the final normalization). At each depth the rows were centred on the corpus mean, scaled to unit length, projected on their 60 leading principal directions (53–68% of their squared length, depending on depth) and varimax-rotated with the rows as indicators, a Q-mode rotation in which the "variables" are tokens-in-context. Each rotated axis was labelled by the fourteen contexts scoring highest on it and the six scoring lowest; the labels are qualitative readings of those lists. Separately, at the same depth, the sentiment arrow (difference of the mean final-token rows of 48 positive and 48 negative sentences) and the sentiment conceptor (fitted to those 96 rows) were computed exactly as in Moore (2026), together with a second conceptor fitted to 16 neutral sentences.

Three quantities were read off. For a unit arrow u and orthonormal axes A, the squared loading on an axis a_j decomposes as (a_j·u)² = ‖A^Tu‖² × (a_j·u)²/‖A^Tu‖²: the share of the arrow captured by the retained subspace, times its concentration on one axis within it. The conceptor's gain along an axis is aCa^T, and the share of its trace inside the subspace is tr(A^TCA)/tr(C). Each statistic is reported against a baseline, because each has a chance level that is not zero. A random unit direction in d = 768 dimensions puts k/d of its squared length into any k orthonormal directions, so capture by 60 directions is 0.078 at chance (0.10 at the 95th percentile) and by 300 directions 0.39 (0.43); the concentration of a random direction on the largest of 60 axes is 0.11 (0.17). A second, stricter null controls for what the arrow is made of rather than for its direction: the sentiment labels of the 96 sentences were permuted 500 times and the difference of means recomputed, giving arrows that are differences of review-sentence final states with no sentiment in them. And because the arrow is read from sentence-final states while the corpus directions are fitted to every token of the novels, the capture statistic was computed a second time with the corpus's own sentence-final states (the 1,133 period tokens) as the reference set.

6.2 Where the concept sits in a layer's leading directions depends on what they are compared with

Three readings of Table 7 are possible, and they do not agree, which is the finding. Read against random-direction chance, the sentiment arrow is at chance in the leading directions of a general corpus at layer 2 (8% against 7.8%) and layer 5 (10% against a 95th percentile of 10%), above chance from layer 8 (15%), and well above it at layers 11 and 12 (34%, 51%); with 300 retained directions the figures are 43%, 52%, 68%, 66% and 72% against a chance level of 39% (Figure 5(a)). That reading says that the concept enters the layer's leading directions only with depth. Read against the label-shuffled null, it says something else. An arrow made of the same sentence-final states with its sentiment removed is captured more than the sentiment arrow at layers 2, 5 and 8 (15%, 17%, 20% against 8%, 10%, 15%), rises with depth in the same way (28% at layer 11, 34% at layer 12), and is exceeded by the sentiment arrow beyond its 95th percentile only at layer 12. Most of the depth trend in the second column is therefore a trend in where sentence-final states of short modern sentences sit relative to the rest of the layer, not in where sentiment sits. The third reading uses the matched reference set, and inverts the first: against the leading directions of the corpus's own sentence-final states the arrow is inside the leading 60 directions at every depth (44–71%), as any difference of sentence-final states is (the shuffled null is 45–58%), and it stands out from that null at one layer only, layer 8, where 71% of it is captured against a null 95th percentile of 62%. Layer 8 is also where Moore (2026) found the positive and negative sentences best separated. The concentration columns tell the same story at the level of single axes: at layer 2 the arrow's largest axis share is at chance whether or not the axes are rotated (0.10 and 0.12 against 0.11), rotation raises it above chance at layers 5 and 8 (0.23, 0.19), and only at layer 12 does one axis carry more than half of the captured arrow (0.54).

What survives is modest and, we think, right. A general corpus does not vary much in sentiment, so sentiment is a low-variance direction of the layer, and an unsupervised reduction of all the layer's states discards most of it at every depth; that is a fact about corpora and about what "leading directions" means, not about where the model keeps the concept. The depth trend that the first reading suggests is carried by a confound the second reading exposes: the sentence-final states of the review sentences move into the corpus's leading directions with depth whether or not they carry sentiment. Once the reference set is matched to the kind of state the arrow is made of, the picture is flat except at layer 8, where the sentiment direction occupies more of the leading sentence-final directions than a relabelled arrow does. That coincidence with the separability result is the one piece of layer-choice information the analysis yields, and Section 7.1 proposes it in that form: a matched reference set, a label-shuffled null, and a layer chosen where the concept stands out from the null, not where its raw capture is highest. The rotated axes themselves remain interpretable at every depth; what they name is the dominant variation in this corpus, literary register, punctuation and syntax, sentence position, and Austen's social vocabulary, and only late in the model is any of it readable as sentiment.

The last layer's most aligned axis is instructive and should be read as a hypothesis from context inspection rather than a mechanism. Its contexts are I wouldn, I don, it doesn, he didn, the stems of negated contractions whose next token is 't. The output layer represents the next token, and at the output layer "negative sentiment" overlaps with "a negation is coming", which is why the sentiment arrow is most captured there and why the capture is not the same thing as the concept. The second-most aligned axis at layer 12 (loading −0.32) is sentiment proper, hopeless, suffering, danger, terrify against nice, right, good. The lesson for layer choice is not "use the last layer"; steering at the last layer is steering the next token directly, which is not the point of a conceptor. The lesson is that presence of the concept in a layer's leading directions, and its alignment with an unsupervised rotation of them, are different properties from separability of the concept, and that the first two depend on the reference set in a way the third does not. A conceptor needs separability. An account of what the conceptor keeps needs the axes, and Section 6.4 shows how to get them at any layer by rotating in the right subspace.

6.3 Reading two conceptors on the same axes

Figure 6 is the payoff of having interpretable axes: a conceptor becomes a labelled profile. The profile is a directional summary: aCa^T is the component the filter keeps along a from an input along a, and a conceptor can also move an input along a into other axes, which the diagonal profile does not show; it describes a conceptor the way the diagonal of a covariance matrix describes a distribution. On the twelve layer-12 axes most aligned with the arrow, the sentiment conceptor keeps 0.86 of the "negation coming" axis and the neutral-sentence conceptor 0.76, so that axis is shared, as it should be, since both sets of example sentences are ordinary clauses that may or may not be negated. On the sentiment-proper axis (51: hopeless, suffering vs nice, good) the sentiment conceptor keeps 0.26 and the neutral conceptor 0.03. On the remaining axes, which are about syntax and punctuation, the sentiment conceptor keeps 0.1–0.25 and the neutral conceptor almost nothing. The comparison between the two conceptors is confounded by their sample sizes (96 rows against 16, hence rank 95 against 15, with the default λ set separately for each), so the neutral conceptor's low gains on most axes say as much about its small sample as about neutrality; the contrast that survives this caveat is between the two axes it does keep and the one it does not. The point of the figure is the form of the description: two 768 × 768 matrices read axis by axis, which is what the conceptor literature's overlap statistics summarize in one number. The gains are modest in absolute terms because most of each conceptor's trace lies outside the 60-axis subspace (19% of the sentiment conceptor's is inside), which is why the next subsection rotates inside the conceptor instead.

6.4 Rotating inside the conceptor's own subspace

A conceptor's eigenvectors span the directions its examples vary along; its top ten, at layer 8, carry gains of 0.95–0.99, contain 99.3% of the sentiment arrow, and account for 17% of the conceptor's trace (the other 83% is spread over the remaining 85 directions at lower gains). What follows describes that leading ten-dimensional subspace, not the whole conceptor. Rotating the ten eigenvectors by varimax, with the 42,000 corpus rows as the indicators, re-expresses the same subspace in axes along which contexts have simple structure. The result is in Figure 7 and Table 9, and the summary has three parts. First, the arrow is mostly one direction in either basis: on the full 96 sentences one eigenvector carries 64% of it and the top rotated axis 73%, and in two fits on 36 of the 48 pairs one eigenvector carries 82–86% and the top rotated axis 68–70%. That eigenvector is the second or third by gain, not the first; the first eigenvector, the direction along which the example sentences vary most, carries 6–10% of the arrow. Rotation does not concentrate the arrow beyond what the eigenbasis already does. Second, the top rotated axis separates held-out sentences by sentiment: fitting the conceptor and the rotation on pairs 1–36 and scoring the 12 held-out pairs, it separates positive from negative with d = 3.6 (the next axis 1.3, no other above 1.0), and on a second split (pairs 25–36 held out) with d = 4.7 (next 2.0). In sample, on all 96, the figures are 3.8 and 1.0.

Third, and this is the comparison that decides what the rotation is for, the same held-out pairs are separated by the arrow itself, the supervised difference of means of the 36 fitting pairs, with d = 7.4 and 6.4 on the two splits; by the arrow's projection into the ten-dimensional subspace with d = 7.0 and 6.2; by the single most aligned eigenvector with d = 4.7 and 5.6; by the first eigenvector with d = 0.4 and 0.8; and by a random direction with mean |d| = 0.6 (95th percentile 1.5), or 0.9 (2.3) for a random direction inside the subspace (Table 8). The rotated axis is well above chance and at the level of the best unrotated eigenvector, and well below the arrow. Rotation inside the subspace therefore does not improve separation, and nothing in it could: the arrow is fitted to the labels and the rotation is not, and an unsupervised axis that happens to align with a supervised direction (cosine 0.83 on both splits) separates about as well as the alignment allows. What the rotation adds is names. The other nine axes are what else the example sentences vary in, and the top axis's own context list (Table 9) shows how weak those names can be: its highest corpus contexts are proper names from Emma and a bench, alongside joy and inexpressibly, and the label "sentiment" rests on the held-out separation statistic, not on the homogeneity of the list.