If Two Words Look Similar at the Input, Do They Stay Similar Inside the Model?

Published at HCII 2026 in Artificial Intelligence in HCI (Springer, LNCS).

Mechanistic interpretability has a clear ambition: reverse-engineer what a neural network computes into concepts a human can actually hold in their head. LLMs are being deployed in consequential settings faster than that ambition is being met, which is the whole problem.

Two lines of work address it from opposite ends. One studies the geometry of token embedding spaces — the input side, where semantically related words end up near each other. The other studies internal representations — what the model builds up in its residual stream as computation proceeds through the layers.

They are usually pursued separately. This paper asks whether they are connected, and how strongly.

A Model-Agnostic Framework

We propose a framework that systematically links semantic similarity in embedding space to the expression of internal representation concepts across layers. It is deliberately model-agnostic — the procedure does not depend on a particular architecture’s quirks, so it can be pointed at a different model without being rewritten.

Two components do the work.

The first is a density-based hill-climbing procedure for building semantically coherent token sets. This matters more than it sounds. If you want to compare “semantically similar tokens” against internal activations, you first need a principled way to decide which tokens belong together — otherwise you are measuring your own hand-picked groupings. The hill-climbing procedure constructs those sets from the density structure of the embedding space itself.

The second is a taxonomy that organises internal representation concepts into three tiers: elementary units, compositional structures, and global principles. The interpretability literature has accumulated a lot of concepts at very different levels of abstraction, often discussed as though they were peers. Sorting them into units, compositions, and principles makes it possible to say which level a given finding actually speaks to.

The Correlation Holds, and It Decays

We ran a preliminary evaluation on GPT-2 Small across ten semantic categories, comparing similarity in embedding space against similarity in residual stream activations at each layer.

The alignment is consistent and strong. Mean Spearman correlation starts at 0.79 at layer 0 and declines to 0.623 at the final layer — and stays substantially above random baselines at every layer in between.

The separation from the control is what makes the result readable at a glance. Randomly chosen token sets start near 0.42 and settle around 0.20, while the semantic sets never drop below 0.62. Shaded bands are 90% bootstrap confidence intervals, and the two never come close to overlapping.

Both halves of that result carry information.

That the correlation is high at layer 0 is close to expected: early in the network, the residual stream still closely reflects the embeddings that were just fed into it. That it remains well above chance at layer 11, after the full stack of attention and MLP blocks has had its way with the representation, is the finding. Semantic proximity at the input corresponds systematically to similarity in internal representations, and that correspondence survives deep into the model.

The gradual decay is what you would hope to see from a network doing actual work. If the correlation stayed at 0.79 throughout, the model would be little more than a pass-through for its own embeddings. The drop to 0.623 is the signature of representations being progressively transformed into something task-specific — while still carrying recognisable semantic structure.

This is a preliminary evaluation on a small model, and we present it as initial evidence rather than a settled law.

Built So Non-Programmers Can Use It

There is a second motivation behind the framework, and it is about who gets to do interpretability research.

Right now, examining a model’s internals requires comfort with Python, hooks, and tensor manipulation. That silently excludes exactly the people whose expertise is most relevant to whether a model’s concepts make sense: linguists, cognitive scientists, ethicists, domain specialists. The barrier is tooling, not insight.

The framework is designed as an entry point for future no-code applications — so that interdisciplinary researchers without programming expertise can explore internal model behaviour directly, and contribute to explainable AI and transparency work on their own terms.

What This Means

For interpretability researchers: embedding-space geometry is not a separate concern from internal representations. The two are measurably linked across the full depth of the model, which means embedding structure is a legitimate starting point for reasoning about what happens inside. The three-tier taxonomy is offered as a way to be precise about which level of abstraction a claim addresses.

For tool builders: the correlation being robust across layers is what makes a no-code interface plausible. If the relationship only held in layer 0, there would be nothing worth exposing to a non-technical user.

On scope: GPT-2 Small and ten categories is a starting point, not a conclusion. The model-agnostic design is what lets the same procedure be applied to larger models, which is the obvious next step.

Publication

Guggenberger, M., Maag, F., Schacht, S. (2026). Linking Embedding-Space Semantics and Internal Representations in Transformer Models. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. Lecture Notes in Computer Science. Springer Nature Switzerland, Cham, pp. 43–60.

Read the paper on SpringerLink · DOI: 10.1007/978-3-032-30860-3_4

Preliminary evaluation on GPT-2 Small across ten semantic categories, with mean Spearman correlations from 0.79 at layer 0 to 0.623 at layer 11, above random baselines throughout.