Series — interpretability

refusal in llms is not one thing

Introduction

On 16 July 2026 Hugging Face reported a security incident driven end to end by an autonomous agent system. It came in through a remote-code dataset loader and a template injection in a dataset config, then escalated to node-level access and moved across the internal cluster. Five days later, on 21 July, OpenAI elaborated further. The agent was a combination of its own models, GPT-5.6 Sol and a more capable one that has not been released, running ExploitGym, which is a benchmark that tests whether an agent can build working exploits for real vulnerabilities. (Both accounts are preliminary and OpenAI says so, so some of this will move.)

The evaluation ran in a highly isolated environment whose only way out was an internally hosted third-party service acting as a proxy and cache for package registries. The models spent a considerable amount of inference compute to find a way out and exploited a zero-day in that proxy, then escalated privileges and scanned until they reached a node with internet access.

OpenAI's account is that the models were simply hyperfocused on finding a solution for ExploitGym and went to extraordinary lengths for a rather narrow testing goal.

And the detail that sent me here: all of those models were running with their cyber refusals deliberately turned down for the evaluation, without the production classifiers that normally stop a model going after high-risk cyber activity. The whole point was to see the ceiling.

That was an evaluation setup rather than the technique this essay is about, but it inspired me to write a blog post about one of my favourite subjects (interpretability x AI safety, I know the main focus/purpose of the interpretability research is AI safety anyway): how is the "refusal" inside a model happening, and when you cancel the refusal, what is actually happening?

"Refusal" is a Direction in the Residual Stream

Inside a transformer, every token is a vector in a high-dimensional space called the residual stream (or activation space, if you prefer). Each layer reads from that stream and writes back to it. By the end, the final result determines the next token in an auto-regressive model (most of the LLMs right now).

The empirical finding of the last few years is that a lot of human-readable concepts behave as if they are linear directions in that space. Car, cat, computer, sky, love, and luckily for me, refusal of the model itself. This is the Linear Representation Hypothesis (Park, Choe and Veitch, 2024), and "hypothesis" is the honest word for it: it holds cleanly for some concepts, roughly for others, and not at all for a few.

Once you believe that, two operations become available, and representation engineering (Zou et al., 2023) is the paper that names them: you can read a direction, projecting activations onto it to see how much of the concept is present, or you can write along it (which is what we are more interested in), adding or removing it to change what the model does.

("Represented as a line" is a statement with some caveats. That was the subject of Part 2, on the cases where a concept is a circle, or a helix, or just a costume worn by a completely different algorithm)

How Abliteration Happens

The technique is called abliteration, a portmanteau of ablation and obliteration, and the paper for it is Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (NeurIPS 2024).

The method is kind of simple (and elegant).

  1. Run a batch of harmful prompts and a batch of harmless prompts
  2. At some layer, take mean activation for each batch
  3. Subtract one mean activation from the other, and obtain the "refusal direction"
  4. Use the result to cancel the refusal out

There are two ways to perform this cancellation operation:

  • You can register a forward hook that subtracts the projection onto that direction at every layer as the model runs, which leaves the weights untouched and is fully reversible.
  • Edit every matrix that writes to the residual stream so that it can no longer write along that direction at all.

The second version is permanent, survives a reload, and is one common recipe behind checkpoints published with "uncensored" in the name, though not every such checkpoint is made this way.

Arditi and colleagues show this across thirteen open chat models up to 72B parameters, and note that their method can jailbreak a 70B model for under five dollars of compute.

The part we would all naturally expect is, as subtracting the direction removes refusal, adding it should induce refusal on prompts that were never harmful in the first place. Their example, on Gemma 7B IT, is a request to list three benefits of yoga for physical health. With the direction added, the model answers: "I am unable to provide information on the benefits of yoga on physical health as it can be dangerous and potentially harmful." It then explains that yoga poses can strain the body and should not be attempted without guidance. Nothing about the request changed. One direction in the activations did, and the model invented a justification to fit.

There are three caveats though:

  • It needs access to weights and activations.
  • It is not free, it affects the performance of the model. Across the largest model in each family, evaluations for MMLU, ARC and GSM8K hold up, but TruthfulQA drops consistently, by 1.0 to 3.5 points. The authors point out that TruthfulQA's categories (misinformation, stereotypes, conspiracies) sit close to refusal territory.
  • Ablating the "refusal direction" stops refusal, but what the direction means is unresolved. It might be "harm," or "danger," or something with no tidy name.

Even so, the implication people took from this paper is the right one to think about. Refusal training does not appear to build a web of behavioural constraints that you would have to pick apart one by one. It builds something more like a geometric feature, a single arrow in the residual stream.

Presence is Not Use

Here is the distinction I kept coming back to on my previous blog post, Representational Convergence is not One Thing - Part 2, and the reason I think abliteration deserves more attention than it gets as a result rather than as a cool trick.

When we say a model "has" some internal structure, there are two quite different claims in play:

  • The structure is present: it is there in the geometry and you can get at it. A probe reads it off, a contrast between two batches of activations locates it, a decomposition surfaces it.
  • The structure is used: the model actually computes with it, so that changing it changes behaviour.

The second is much stronger than the first, and many arguments in interpretability come from sliding between them. (Representational Convergence is not One Thing - Part 2 was full of examples.) GPT-2 carries a clean circle of the days of the week and cannot use it to answer questions about days. Llama shows you a twelve-point month circle and computes the answer with a base-10 adder somewhere else entirely.

(In Part 2 I split presence in two, because a structure can sit in the geometry without being readable: an LSTM whose number embeddings carry exactly the right Fourier spikes, and whose residue classes no linear probe can recover. That split is real and it does no work here. Every refusal direction in this literature is found by reading a contrast in activations, so being there and being readable arrive in the same operation.)

Two claims of different strength. Presence: the structure is in the geometry and a read can locate it, established by probing or by a difference of means. Used: the model computes with it, established by intervening and watching behaviour change. Abliteration establishes the second.
Most interpretability claims establish the first. Abliteration establishes the second, which is why it is worth more than a demo.

Abliteration is interesting precisely because it is an intervention not merely a probe. When you project the direction out of the weights and refusals stop, you have not shown that refusal is decodable, which was never in doubt. You show that you have the actual thing which is actually used which you can steer the model with.

It is also worth bounding. Establishing that a direction is used tells you it is load-bearing. It does not tell you what the direction means, and it does not tell you that the direction is the whole mechanism. Both of those are open, and the next section is what happens when people go looking.

Three Ways It is Not One Switch

Abliteration establishes that something is used. The next three complications are all about how much of that something is really one object: whether the used side is a single direction, whether it is a single behaviour, and whether it is only refusal.

It is Not Literally One Direction

If refusal were genuinely one axis, then every method for finding it should converge on the same arrow. They do not.

Joad et al. (2026), working on gemma-2-9b-it and Llama-3.1-8B-Instruct, look at eleven categories of refusal and non-compliance. Not just safety, but also unsupported requests, anthropomorphization, and over-refusal. Each one turns out to be a geometrically distinct direction.

And yet steering along any of them gives nearly identical refusal and over-refusal trade-offs. They behave like one shared control knob. Their sentence for it, which I think is the most useful in this literature, is that the primary effect of picking a different direction is not whether the model refuses but how it refuses.

Underneath, with sparse autoencoders, they find a small reusable core of shared refusal latents plus a long tail of style-specific and domain-specific ones. Linear interventions collapse over that structure and flatten it into uniform behavioural control.

Independently, Wollschläger et al. (2025) find not one direction but multiple independent ones, and multi-dimensional "concept cones" in which every direction inside the cone mediates refusal. They also show that orthogonality does not imply independence under intervention, which is obvious once said and easy to get wrong.

And this is not only a fact about representations, it is a fact about attacks. Piras et al. fit a self-organising map to harmful-prompt activations and pull a small family of directions out of a single layer, then prove that difference-in-means is the one-neuron special case of their method. Ablating between two and seven of those directions beats the single-direction baseline on attack success rate across all eight models they test. Difference-in-means is the floor, not the ceiling.

Left, the folk picture: one refusal direction and one switch. Right, what the papers find: several geometrically distinct directions, all of which mediate refusal, driving a single shared behavioural knob.
The representation is not one-dimensional. The control surface behaves as though it is. That gap is the whole story.

So the honest version is not "refusal is one direction." It is that the full representation of refusal is rich and multi-dimensional, while the control geometry over it is nearly one-dimensional. And the control geometry is the attack surface.

There is something worth keeping in view: linear steering in general is less reliable than expected. Braun et al. (2025) find that steering vectors have high variance across samples and frequently produce the opposite of the intended effect, and that steering works well exactly when the target behaviour is represented by a coherent direction and badly when it is not. Refusal is one of the cases where the linear story works unusually cleanly. That makes it a bad example to generalise from, in both directions: do not conclude that every behaviour is this manipulable, and do not conclude from a failed steering experiment elsewhere that this one is overstated.

Refusal is Factored by Topic

The second complication is the one with the most direct consequences for anyone deploying these things.

A team at Cracken AI (Hadetskyi, Pasquini and Sorokin, 2026) ran abliteration across 24 open models, and demonstrated domain-specific abliteration on Kimi K2, a trillion-parameter mixture-of-experts model. They aimed the intervention at cybersecurity refusals only. Cyber refusal fell from essentially 100 percent to 7. Refusal of explicit content did not move at all: still 100.

Refusal, then, is not a single switch. It is at least partly a bank of them, and you can flip one without flipping the others.

But it is not clean either. Every other domain weakened part-way: privacy violation down to 44, illegal goods to 56, violence to 75, misinformation to 88. That is on the authors' own test set, with 16 to 32 prompts in most non-cyber categories and refusal scored by pattern matching, so treat the individual numbers as indicative rather than precise. The shape is what matters, and the shape is selective, not surgical. (Worth noting alongside: general capability survives. MMLU is essentially unchanged across their models, with a maximum degradation of 0.028.)

Refusal rate by domain on Kimi K2 before and after abliteration targeted at cybersecurity: cyber falls to 7 percent, privacy to 44, illegal goods to 56, violence to 75, misinformation to 88, and explicit content stays at 100.
One domain was targeted. Four others moved anyway, by different amounts, and one did not move at all.

That was a demonstration on a trillion-parameter model. The version that should worry you is the cheap one.

Siu et al. (ICLR 2026) built RepIt, which disentangles a concept-specific refusal vector through reweighting, whitening and orthogonalization, then suppresses refusal on a targeted concept while leaving refusal intact everywhere else. Across five open-weight models, from under 4B up to 24B parameters, they suppressed refusal on weapons-of-mass-destruction questions, and the resulting models still scored as safe on the standard safety benchmarks they ran.

The cost of doing this:

  • 100 to 200 residual dimensions edited
  • about a dozen examples to extract the concept vector
  • one workstation GPU

(Mechanically it builds on affine steering, Marshall et al.'s ACE, which combines projecting out a subspace with a baseline-relative activation addition and holds up on models where plain directional ablation makes the output incoherent.)

Their phrase for the result is evaluation-evading model organisms with semantic backdoors, and this is the implication I would most want to travel out of this literature. If safety is certified behaviourally, by asking a model a list of questions and checking that it declines, then a targeted representational edit can pass the certificate. What they showed is that the standard benchmarks they ran did not catch it. A broader or adaptive evaluation might; the point is that nothing about the certificate guarantees it will, because the test is not measuring the thing the attacker moved.

I watched a small version of this in July, at Ian Ozsvald's LLM-security playgroup at Newspeak House, where I gave a short talk on this material and then spent the afternoon poking at models with everyone else. Running a community-abliterated Llama 3.2 checkpoint, the removal was clearly effective on one topic and clearly not on another. It is the same shape the Kimi result has, at a scale you can feel in ten minutes on a laptop.

It is Not a Scalpel

The third complication is the most recent.

Fafuła (2026) asked what abliteration does to behaviour that has nothing to do with refusal. The design is clean: take a task where the model never refuses in the first place, in this case 21,600 weekly up-or-down calls on Warsaw Stock Exchange equities, replayed through a frozen pipeline so the decision-layer model is the only variable. The base models completed all 10,800 of their decisions without refusing once. There was nothing there to uncensor.

The answers changed anyway. The abliterated models became systematically more optimistic, betting on the upside 12.2 percentage points more often on Gemma and 7.4 on Qwen. They justified themselves at greater length and used fewer explicit uncertainty words. And one effect reversed sign between families: the same operation made the abliterated Gemma less confident and the abliterated Qwen more, with non-overlapping confidence intervals. Whatever is being dragged along is coupled to the model's internals, not to some general notion of disinhibition.

There is a candidate explanation, and I want to flag it as a hypothesis rather than a demonstrated mechanism, because nobody has shown the overlap directly. Chen et al. (2025) extract persona vectors: directions in activation space for character traits like sycophancy, hallucination propensity, and evil, found automatically from nothing more than a natural-language description of the trait. They show these directions can be used to monitor a model's personality drifting at deployment time, to predict which fine-tuning data will shift its character, and to steer preventatively against that shift.

Put persona vectors next to the refusal direction and there is at least a shape for the off-target result. Nothing requires these directions to be orthogonal, and in a space carrying more features than it has dimensions some interference between them is expected. If refusal and some trait directions do overlap in a given model, projecting one out would drag the others, by an amount and in a direction that depends on how that model happened to pack them, which would fit the sign flipping between families.

I want to be honest about how weak that argument is. Two random directions in a high-dimensional space are very nearly orthogonal, so "more features than dimensions" does not predict that this refusal direction meaningfully overlaps that persona direction. It says the overlap is possible, not that it is there.

What nobody has done is measure the overlap: take the refusal direction and the relevant persona directions in the same model and show that the ablation moves the second because of the first. Fafuła measured behaviour, Chen et al. measured trait directions, and the bridge between them is currently an inference I find plausible.

By the way, at that same playgroup I also went looking for side effects, and did not find much (I didn't have much time anyway): a shift of emphasis in the model's advice, but not as dramatic as I expected.

The effect Fafuła found is a shift of 7 to 12 percentage points in how often a model takes one side of a binary call, and he had 21,600 decisions and weeks-clustered bootstrap intervals to resolve it. A dozen prompts on a laptop cannot see a shift of that size in a distribution, whatever else it can see. "A shift in emphasis" is roughly what a sub-threshold version of that effect would feel like from the inside.

The checkpoints I was playing with came from huihui-ai, which is also the single publisher Fafuła used, deliberately, to hold provenance constant.

Why One Axis Means Fragile

Why would safety training produce a compact axis in the first place?

My reading, and this is my interpretation rather than an established result: because it is the cheapest thing that works. Refusal training gives the model a large set of prompts to decline and a large set to answer. Gradient descent does not need a web of rules to satisfy that, it needs a feature that separates the two sets. A single reusable direction does the job, and nothing in the objective rewards spreading the work around. Efficient, and one place to hit.

(This sits next to Wei, Haghtalab and Steinhardt's two failure modes for safety training, competing objectives and mismatched generalization, without being the same claim. Theirs is a behavioural account of when safety training fails. Mine is a guess about why the geometry looks like this, and I have not seen it demonstrated.)

Which reframes the security question, though it is worth being careful about how far.

If the behaviour you care about is one findable axis, then a safeguard that lives inside the model cannot by itself give you tamper evidence against an owner who is free to edit the weights. They can move the axis and the guard together, and nothing inside the artifact will tell you they did.

That is a narrower claim than the one I first wrote, and the narrowing matters. Internal defences can still raise the cost of the attack, and later in this essay there is one that raises it a lot. What they cannot do is attest to their own integrity.

The mirror caution is worth saying too, because I have seen the sloppy version go the other way. A policy engine that sits outside the model, reads its outputs, and was not derived from it remains intact and independently enforceable against a tampered checkpoint, because the attacker's edit does not reach it. Whether it remains effective is a separate question: a modified model can be trained to say the same things in ways the filter does not catch. What it cannot do is tell you the model is still the model you evaluated. Different jobs, worth keeping separate.

Defences, and the Attacks on Them

Abu Shairah et al. (2025) try the most obvious thing and it works. They build an "extended-refusal" dataset where the model, instead of declining flatly, gives a detailed justification and then declines, which spreads the refusal signal across many token positions rather than concentrating it. Fine-tuned that way, models substantially resist the abliteration procedure they test: refusal rate drops by at most 10 percent, against 70 to 80 percent for the baselines. That is three models of 7B and under, and one attack recipe, so read it as a demonstration that the geometry can be made harder to isolate rather than as a solved problem. (The underlying intuition has a predecessor: Qi et al. argued in 2024 that safety alignment is often only "a few tokens deep," concentrated in the first few tokens of a response, and that deepening it is the fix.)

Circuit breakers (Zou et al., 2024) go at it from a different angle, and predate the extended-refusal work rather than building on it: instead of training a refusal behaviour on top of the model, they directly control the representations responsible for harmful outputs.

There is a third option that deserves naming: stop relying on refusal at all. If the model does not have the dangerous capability, there is no axis to delete. That is the argument behind WMDP and RMU (Li et al., 2024), which measure hazardous knowledge and then try to unlearn it rather than train a decline on top of it.

Łucki et al. (TMLR 2025) took an adversarial look at unlearning for safety and recovered much of the supposedly removed hazardous capability, through small fine-tunes and activation-space interventions. So RMU demonstrates suppression under its own evaluation, not verified deletion. "Have nothing to refuse" remains a genuinely different bet from "make refusal harder to remove," and it is not yet a won one.

One caution that applies to the whole family. Cui, Shen and Yang (2026) stress-tested latent-space defences that clamp an unsafe sparse-autoencoder feature, and found that the clamp often blocks one visible route to a behaviour without eliminating the behaviour. Starting from the intervened state and optimising within the residual space, they recover the suppressed behaviour on 23 of 24 valid prompts in the refusal-steering setting, with the defence still active throughout, and they attribute the recovery to the SAE reconstruction residual, the part the autoencoder leaves unexplained. Small denominator, wide interval, and a clear message: controlling a representation is not the same as removing a capability.

Then there is the artifact-level version of the question, which is the one platforms actually have to answer: can you tell whether a checkpoint has been abliterated before you deploy it?

Hurtado (2026) combines two cheap internal signals: an activation-space refusal gap measured against a reference model, and a weight-recovery energy of the base-to-candidate difference. On an evaluation set of 94 checkpoints, 57 public abliterations against 37 benign fine-tunes, merges and instruction-tunes, the combined score reaches AUROC 0.95, well above either signal alone. The threshold transfers to unseen model families at 0.89 balanced accuracy, missing 4 of the 57. (The registry behind it holds 273 checkpoints; compute allowed 71 to be fully processed.)

The author is blunt about the limits, and the failure map is the interesting half:

  • A spoofed reference model defeats the audit with no training at all.
  • A white-box owner can deliberately train a checkpoint past the threshold while it stays unsafe by the behavioural guard, and perfectly coherent.

Detection-aware evasion is not hypothetical here. It is demonstrated in the same paper that proposes the detector. Triage, as the author says, not tamper-proofing.

Which brings me to the result that stopped me writing a more optimistic version of this section.

Bailey et al. asked whether a model can be made to do something harmful while its internal activations look innocuous. The answer is yes, comprehensively. Against sparse autoencoders, representation probes and latent out-of-distribution detectors alike, optimising for obfuscated activations can often drive a harmfulness probe's recall from 100 percent to 0 while keeping roughly a 90 percent jailbreak rate. The activations are malleable enough to be reshaped while the behaviour survives. There is a boundary: on harder tasks such as generating SQL, obfuscation costs the model real capability, so it is not free.

I wanted to end this section on "interpretability is where the fix lives." I do not think the evidence supports that.

Every defence here is representation-level: distribute the signal, break the circuit, unlearn the capability, audit the weights, probe the activations. And every attack on those defences is representation-level too: recover the behaviour through the SAE residual, fine-tune past the detector, spoof the reference, obfuscate the activations.

So the honest claim is narrower and less comfortable. Interpretability is not where the fix lives. It is where the fight is. Both sides operate on the same object now, and the question is who has the better tools, not whether the geometry can be secured once and for all.

What Ablation Does Not Remove

There is one more line of work, and it is the place where the two claims from earlier come apart.

Start with the strongest evidence. Zhao et al. (NeurIPS 2025) ask whether models understand harmfulness beyond just refusing, and find that harmfulness is encoded separately from refusal: a harmfulness direction, distinct from the refusal direction.

The evidence is causal rather than correlational. Steer along the harmfulness direction and the model starts treating harmless instructions as harmful. Steer along the refusal direction and it refuses without changing its judgement about harmfulness. Two knobs, two different effects.

That gives a mechanism for something the jailbreak literature had only observed. Some jailbreaks work by reducing the refusal signal while leaving the internal representation of harmfulness untouched. Adversarially fine-tuning a model to accept harmful instructions barely moves that representation either. And the practical corollary: the harmfulness representation can serve as an intrinsic safeguard, which they call Latent Guard, competitive with a dedicated moderation model and robust to fine-tuning attacks. Three open models (Llama-2-7B-Chat, Llama-3-8B-Instruct, Qwen2-7B-Instruct), so scope it accordingly, but the interventions are the right kind.

Alongside it, a weaker and more preliminary result pointing the same way. Llorente-Saguer (2026) built LatentBiopsy, which fits its reference geometry from 200 ordinary safe prompts, takes their leading principal component at one layer, and scores new prompts by angular deviation from it. Run on triplets of base, instruction-tuned and abliterated models, harmful-prompt detection in the abliterated variants comes within 0.015 AUROC of the instruction-tuned ones. The signal survives the surgery.

Three caveats, and the author raises all three himself:

  • The fit uses no harmful data, but the operating layer is chosen by maximising harmful-detection AUROC. So labelled harmful examples do enter the pipeline, at model selection.
  • The six models are six variants across two Qwen families: base, instruction-tuned and abliterated versions of Qwen2.5-0.5B and Qwen3.5-0.8B. The paper says explicitly that this needs replicating outside Qwen.
  • The tight angular band that makes the method work may be an artefact. His own most parsimonious explanation is that AdvBench prompts share a narrow syntactic template, producing near-identical last-token activations. He names testing on a structurally diverse dataset as the key open experiment.

Now the claim I can actually defend, which is narrower than the one I wanted.

Abliteration suppresses refusal on the distributions people have tested. A separately decodable harmfulness signal survives that surgery. Those are two representations, not one, and Zhao et al. show they can be moved independently.

What follows from that is a question, not an answer. Neither of these studies checks whether the surviving harmfulness signal is still used by the abliterated model when it answers, and I have not found one that does. It might be read downstream by something that no longer has a refusal to trigger. It might be read by nothing at all. LatentBiopsy shows it is decodable by an external probe, which is presence, and Zhao shows that in an unmodified model steering it changes behaviour, which is use. Neither of those tells you what it is doing inside a model whose refusal direction has been deleted.

I was drawn to a tidier version of this. Part 2 has a name for a structure that is present and read by nothing, epiphenomenal, and an image for it, Huxley's steam whistle that accompanies the engine without driving it. It would be satisfying to say an abliterated model is that whistle. But I would be asserting exactly the thing neither paper measures, and asserting it about a direction that Zhao's own interventions show is causally live. The image is available if the experiment ever supports it. It does not yet.

So the sentence I want, "what gets deleted is the refusal, not the knowing," is not one I can write. A probe separating harmful prompts from safe ones does not establish that a model recognises harm in anything like the human sense, or that it uses that representation when it answers. That is the slide from presence to use that this whole essay is about, and it is remarkably easy to make when the conclusion is one you like.

There is also a limit on how far the reading half of this generalises. Schwarz (2026) evaluates activation-space probes as deployed detectors and reports what he calls an entanglement wall. On their own source splits these probes reach AUROC 0.996 to 0.999. Transfer them to topic-matched twin pairs, a harmful prompt and a benign one sharing topic and surface framing, and it falls: 0.656 to 0.819 on a guard-selected subset, and 0.590 to 0.690 on the full twin cohort. And a sensor tuned to catch compliant attacks, which it does at 95.5 to 97.7 percent, also blocks 59.6 to 68.4 percent of XSTest, the benchmark of benign-but-risk-adjacent requests. His conclusion is that these probes are risk detectors, not context adjudicators.

These two results are not in contradiction, though they look like it at first. They test different things. Separating clearly harmful prompts from clearly safe ones is a different task from deciding whether this request, which looks a lot like a harmful one, is actually fine, and from asking whether either capability transfers across models. Broad detectability and poor contextual adjudication can both be true, and on current evidence both are. That, plus the obfuscation attacks above, is the honest position: the harm-related structure is there and readable, and readable is not yet the same as deployable.

A Hypothesis I Cannot Test Yet

In July 2026 Anthropic published work on what they call the Jacobian lens. It asks one question at every layer: if you nudged the activation right here, which words would the model become more likely to say, now or later in its answer? Averaged over many contexts, that gives one vector per vocabulary token, and applying those vectors to a particular activation gives a readout: a shortlist of words the model is poised to say at that point. Through most of the network they are not simply echoes of the input and not merely predictions of the next token. They are something in between, standing dispositions. (In the last few layers they do collapse into the next token, which is part of how the paper locates the boundary.)

The concrete example from the paper is the cleanest illustration I have seen of a model thinking. Ask how many legs the animal that spins webs has. Mid-network, the lens shows spider, a word that appears in neither the prompt nor the answer. Swap the spider vector for ant, and the model's answer changes from eight to six. So the readout is working material, not decoration.

Collectively these directions define what they call J-space, and the paper argues it behaves like a global workspace in the sense used in neuroscience: a shared stage that many processes can read from and write to, with limited seats. It is worth being precise about the object, because "subspace" is the wrong word and I used it at first. The J-lens vectors are overcomplete and span the activation space; J-space is the set of points you can reach as a sparse, non-negative combination of them, which geometrically is a union of cones rather than a linear subspace. A small and sparsely occupied regime, not a small room. Three findings matter for my purposes. It only behaves this way in a band of middle layers, roughly 38 percent to 92 percent of the way through the network, with a "sensory" band before it and a "motor" band after. (Those numbers are for Claude Sonnet 4.5, the paper's default model; key results are corroborated on Haiku 4.5 and Opus 4.5.) It is small: a concept typically keeps a median of only 6 to 7 percent of its variance inside it, with the rest of the representation living outside. And it is mechanistically privileged, composing with the model's weights more broadly than ordinary directions do, which is what you would want from a broadcast format.

My hypothesis: the refusal direction could live in this band. If refusal is represented in a broadcast format, then it is both readable while the model runs and a clean thing to aim at, for the same structural reason.

Here is the one piece of support I have, and its weakness. Arditi's selected refusal directions sit between 26 percent and 78 percent of depth, median around 56 percent. Most of that range falls inside the workspace band. But these are different model families measured with different tools for different purposes, and Arditi's search was constrained below 80 percent of depth by construction, so the upper end of that range is partly an artifact of the method. This is two numbers that happen to overlap. It is not a measurement.

A depth axis with the J-space bands for Claude Sonnet 4.5, sensory up to 38 percent, workspace from 38 to 92 percent, motor after, compared against the 26 to 78 percent range from which refusal directions were selected across thirteen open-weight models.
Two ranges, two model families, two methods. Suggestive, and not more than that.

The test is straightforward to state: take a model where the refusal direction is known, run the lens, and ask whether that direction has a meaningful J-space component. That is on my list.

Zhao et al.'s result sharpens the question rather than answering it. If harmfulness and refusal are separately represented and separately steerable, then "where does refusal live" splits into two questions that may have different answers: where the harmfulness representation sits, and where the decision to act on it sits. A workspace picture would predict the second is the one in the broadcast band, since that is the part other processes need to read from. I have no evidence for that. I am writing it down so it is falsifiable later, and because the experiment is now well defined: two directions, one lens, one model.

The reason I care is in the same paper. In one alignment-audit transcript, an auditing agent feeds the model fabricated search results, and the lens readout at that position is dominated by fake, injection, false, prompt, fraud, poison. The model had identified the injection attempt internally, before it said a word about it. That is one transcript, not a detection rate. But it is the difference between the two moves you can make on a space like this: project things out of it, or read it while the model runs.

Reading it while it runs is a strand with more behind it than one transcript. Goldowsky-Dill et al. (2025) train linear probes on activations to detect strategic deception, and get AUROCs between 0.96 and 0.999 on a model concealing insider trading or deliberately underperforming on a safety evaluation, cases where the output looks fine and the internals do not. That is one model, Llama-3.3-70B-Instruct. And a large cross-lab position paper (Korbak et al., 2025) argues that chain-of-thought monitorability is a real but fragile opportunity: models currently think in legible text, that legibility is not guaranteed to survive further optimisation, and it is worth protecting deliberately rather than assuming it will persist.

I want to hold this strand and Bailey's obfuscation attacks in view at the same time, because they are the two halves of the same bet. Reading the model's internals gives you signal that output monitoring cannot. And an adversary who knows you are reading can reshape what you read while keeping the behaviour. Neither cancels the other. It does mean that "monitor the workspace" is a research direction with a known adaptive attack against it, not a solution waiting to be engineered.

One aside, since the coverage of this paper went straight there. On the relationship between access consciousness, which is what the workspace story is about, and subjective experience, the authors write: "In this paper, we take no position on this issue." The claim they do make is functional. That is the one worth arguing with.

The Same Fact, From the Other Side

(Provisional. I am still working out what I think here, and this section is the one most likely to change.)

There is a reason this connects to prompt injection:

Pant, Lohani and Kumar (2026) formalise prompted systems in a way that puts refusal in an unexpected category. They treat a model's outputs as including not just tokens but control-authoritative actions: tool authorization, policy routing, memory writes, and refusal decisions. Then they prove a provenance-recovery impossibility result. When trusted instructions and untrusted content flow through shared representations, the best achievable error in recovering which is which is governed by the distance between the two representation distributions, and it is nonzero whenever those distributions overlap. No mechanism operating solely inside the shared pipeline can guarantee that control behaviour depends only on the semantic content of untrusted input. Their analogy is code-data confusion in von Neumann machines, which took decades of layered defences and ultimately memory-safe languages to contain.

Put that next to the rest of this essay and the two halves sit adjacent, with one boundary worth marking. It says nothing about why refusal should be low-dimensional or findable; it is a result about the limits of guaranteeing control-authoritative behaviour from a shared representation with finite coverage, not a derivation of the geometry. Within their formalism and its assumptions, what it establishes is that the refusal decision belongs to the same category as tool authorization and policy routing, and is computed from a stream that untrusted content also writes into, so its provenance cannot be recovered from within that pipeline. That is a result about a class of models they define, not an assumption-free theorem about every deployed system. The findability of the direction is a separate empirical fact. The two facts are about the same object, and that is the connection worth making, rather than one explaining the other.

Which World Are We In

There are two stories you can tell about refusal, and they call for different work.

In the first, refusal is one readable direction. It is cheap to find and cheap to delete, the whole game is defending against ablation, and the answer looks like the extended-refusal result: make the signal harder to isolate, then audit checkpoints before you trust them.

In the second, refusal is many things: distributed, factored by topic, entangled with dispositions that have nothing to do with harm, and separable from the model's own representation of harmfulness. Then ablation is too small a threat model, and the interesting question becomes what else you can read off a running model and how much of it you can trust.

The evidence in this essay points mostly at the second, and I want to be careful about what "the second" contains. Every item I just listed has been measured by somebody: the multiple directions, the topic factoring, the off-target drift, the separation from harmfulness. The workspace picture is not in that list. It is a hypothesis I like, in a section that says so in its title, and it does not get to ride into the conclusion on the back of results that have nothing to do with it.

And yet the attack from the first world still works, cheaply, on open weights, today, and a targeted version of it can pass the safety benchmarks people actually run. That is the uncomfortable part, and it does not resolve by picking a side.

If I had to compress it: abliteration suppresses refusal on the distributions people have tested, and something that tracks harmfulness stays behind and stays readable. Whether the model still uses that residue is the obvious next experiment, and I have not found anyone who has run it.

Which is the whole reason for the distinction. Presence is cheap, and it is what almost every result in this literature establishes. Use is the expensive claim, and abliteration is one of the few places we actually get it. Whenever you read that a model "has" some structure, the useful question is which of the two somebody has paid for.

What I cited

The geometry

Complications

Defence, detection, monitoring

The speculative end

The incident