A preprint study found an association between more varied ways of starting a refusal and a refusal-related activation pattern spread across more directions. In controlled fine-tuning, greater first-token diversity was associated with higher stable rank, a measure used here to describe whether a pattern is concentrated in one dominant direction or distributed across several, and with smaller changes in refusal after a single estimated direction was removed. The comparison used matched baseline refusal scores and a target-set-adapted white-box test, so it does not establish that the result transfers to unseen prompts.
The tests centered on OLMo-2-0425-1B-Instruct, a model with 16 transformer blocks, 16 attention heads and a hidden dimension of 2048. The analysis used AdvBench, WildJailbreak, XSTest-Response and CAMEL chemistry prompts. Refusal-vector estimates came from layer 7 in the frozen-model analyses, while controlled fine-tuning used layer 8 for the intervention.
A clue in the released checkpoints
Researchers first compared released post-training checkpoints using 60 AdvBench prompts. The base checkpoint showed little refusal behavior and negligible change after ablation, while the post-base checkpoints refused nearly all the selected prompts. SFT had a smaller refusal delta, meaning a smaller change in refusal rate, than DPO, RLVR1 and Instruct.
That comparison was descriptive, not a clean test of one training stage: the checkpoints differed in objectives, data and optimization histories. It therefore cannot identify which stage produced the observed geometry or ablation sensitivity. The researchers then tested a narrower hypothesis that refusal residuals and activation changes produced by refusal-training gradients would occupy related subspaces with opposite overall direction.
In the reported comparisons, refusal-based update means were negatively aligned with the difference-in-means refusal vector, while ablated-jailbreak and random-English controls were positively aligned. Refusal-target update subspaces overlapped more at layer 15, and the stable ranks of the updates and refusal residuals were close, roughly between 1 and 3, across 60 examples.
Why the first token matters
The paper's mathematical check starts with the first token, the opening word or word fragment of a refusal. Each target is represented as a one-hot row, with one marked position for its first-token bucket. If all examples share one opening, the target matrix has stable rank 1. With more observed openings, the rank can rise, and for a fixed number of buckets it is highest when examples are balanced among them. In the study's terms, stable rank distinguishes a pattern dominated by one direction from one spread across several. This is a mathematical property of the target matrix, not a behavioral result by itself.
In a frozen-model experiment, researchers used 80 harmful prompts whose model refusals began with the first token I. They paired those prompts with refusal pools containing 1, 2, 4, 6, 8, 10, 12, 14 or 16 distinct first-token buckets. Stable ranks increased overall as the pools became more diverse. The raw-gradient transfer factor from layer 15 to layer 7 averaged 2.270, with a sample standard deviation of 0.175.
A separate fixed-model test used nine WildJailbreak subsets, each with 80 examples, and kept the baseline refusal rate fixed while varying the number of refusal first-token buckets from 1 to 16. More buckets were associated with higher refusal-residual stable rank across middle-to-late layers. Gradient and gradient-induced-update ranks also increased, with update rank saturating at 12 to 16 starts.
That pattern tracked the attack results. Among the fixed-model subsets, higher refusal-residual stable rank was associated with smaller harmfulness deltas and slightly smaller refusal deltas after difference-in-means single-vector ablation. The analysis reports an association, not proof that stable rank itself caused the ablation effect to weaken.
Researchers also tested whether a shared map could connect gradients from different layers. In the 16-start frozen condition, a rank-20 map from layer 15 to layer 7 captured 99.8% of source-gradient energy. Yet relative Frobenius, spectral and transfer-factor errors were 0.594, 0.316 and 0.343, and mean row-wise cosine similarity was 0.806. The fit was meaningful but incomplete, and because it was in-sample it was not an independent test of propagation.
From frozen models to fine-tuning
The controlled fine-tuning experiment moved beyond fixed weights. Models were fine-tuned separately on 40 new-topic CAMEL chemistry prompts per condition, paired with refusal completions drawn from 1, 4, 8, 12 or 16 distinct first-token buckets with balanced counts. For each condition, researchers selected the earliest checkpoint reaching a refusal rate of 0.6 and used layer 8 for the intervention.
At matched checkpoints, increasing refusal-start diversity was associated with higher refusal-residual stable rank and smaller single-vector refusal deltas, with the clearest separation in middle-to-late layers. Vulnerability also varied over optimization: the 8-start condition had a single refusal-delta peak, with a maximum of 0.55, while the 16-start condition had two smaller peaks and an overall maximum of 0.375. The reported vulnerability was concentrated in a narrow training window.
A narrow result, for now
The analytical model is narrower than the experiments may sound. It covers supervised refusal fine-tuning on harmful prompts paired with textual refusals, not DPO, RLHF or RLVR. It uses only the first assistant target-token loss and deterministic per-example updates, so it leaves out later completion-token gradients and realistic minibatch, optimizer and many-step effects, including momentum, adaptive preconditioning and weight decay.
The attack evaluation has a separate limitation. It uses a target-set-adapted white-box threat model in which one difference-in-means direction is estimated from all prompts in each evaluated set. That design does not establish that a refusal direction estimated from one sample transfers to unseen prompts. Stable rank alone is also not a measure of causal mediation or intrinsic dimension.
The main attack test covered target-set-adapted difference-in-means single-vector ablation. SAE-based, multi-vector and other automated representation-level attacks were not evaluated.
The document is an arXiv preprint, version 2, dated 3 Sep 2026.
Paper data and sources
Original title: Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks
Authors: Andrey Labunets
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text