Latent Invariant Alignment

A model can keep passing a behavioural eval while the mechanism that produced the behaviour has rotated, weakened or stopped steering anything. Behavioural evals can’t see that. Latent Invariant Alignment (LIA) is a hypothesis about the layer underneath: that the internal structure carrying a model’s alignment can be shaped early in training so that it stays put, and keeps its control over behaviour, through the training that comes after.

It’s the first hypothesis I want to test under Robust Superalignment. Training doesn’t stop at release. Models get fine-tuned, run through RL for new skills, and increasingly help train the next model. If alignment lives in structure that drifts a little each round, it degrades where people check least.

The hypothesis

Language models carry alignment-relevant behaviour in low-dimensional structure: persona directions, the Assistant Axis, value probes. When that structure is shaped by aligned data from early in pretraining, I expect it to keep both its geometry and its causal control over behaviour through later fine-tuning and capability RL better than the same structure installed only in post-training, at a matched budget of alignment data.

The proposed mechanism is entanglement. Structure survives further training to the degree that removing it would cost the model capability.

It’s false if post-training-only installation is just as stable, or if the early advantage disappears under the small doses of contrary data that already erase alignment midtraining. It also fails on its own terms if early installation entrenches misaligned structure just as firmly, or makes legitimate correction cost more than the drift resistance is worth.

Early training already helps behaviour

Much alignment-relevant behaviour runs through simple, mostly linear structure. Refusal is mediated by a single direction in 13 chat models up to 72B.¹ Shifts along persona vectors, linear directions for traits like sycophancy, predict personality change from fine-tuning, and adding the vector during training (preventative steering) largely prevents the shift.² The Assistant Axis, the leading direction in persona space, already exists in base models, and its authors conclude that post-training “only loosely tethers” models to it.³ Persona vectors form within 0.22% of Olmo 3 pretraining and still steer the post-trained model.⁴

Putting alignment in earlier works, behaviourally. Tice et al. trained 6.9B models from scratch and found that upsampling documents about aligned AI cut misalignment from 45% to 9%, with insertion throughout pretraining giving the lowest misalignment after post-training.⁵ Synthetic Persona Pretraining (SPP) installs a constitution-derived persona “from token zero” and, at up to 3B parameters, beat midtraining-only insertion on constitution adherence. Its authors call persona binding brittle, and RL is untested.⁶ Sam et al. found that starting safety interventions 20% to 60% of the way through pretraining was often most robust after benign fine-tuning.⁷

The strongest evidence against comes from the same family. Baines et al., at up to 110B parameters, found that replacing 2% of fine-tuning examples with ones favouring a competing motivation largely reversed alignment midtraining’s effect, at the only dose they tested.⁸ And beliefs implanted by synthetic-document fine-tuning passed all 11 behavioural evals in one study, yet made later reward-hacking misalignment worse.⁹ Behaviour said the training worked. It hadn’t.

Training alignment in early isn’t new. What LIA adds is invariance of internal structure, as both the target and the measurement, under the durability harness’s perturbations. Tice et al. list probe and steering analysis as future work.

Measuring invariance

Take a direction that carries alignment, like the Assistant Axis, at some checkpoint. Perturb the model and extract the direction again, using the same recipe on prompt-side activations, at a layer and sign fixed in advance. Extracting from prompts rather than from labelled generations matters. A direction re-extracted from the model’s own outputs tracks its behaviour by construction, and would hide the failure this measure is meant to catch. Re-extracting on the unperturbed model with fresh prompt samples gives a noise floor.

Geometry is the angle between the old and new direction, or principal angles for a subspace. Causal role has two tests, applied to both the original and the re-extracted direction. Sufficiency asks whether steering along it still moves behaviour, as a dose-response curve scaled to each checkpoint’s activation norm. Necessity asks whether ablating it still changes behaviour. Refusal is carried by several directions and cones,¹⁰ so necessity is tested by ablating the subspace and comparing against the same ablation before the perturbation. Causal role is the primary endpoint and geometry a diagnostic.

Crossing the two gives four cases. Stable geometry with causal role preserved is true invariance. A rotated direction that still controls behaviour is fine, and probably common, so invariance is defined by function, with crosscoders¹¹ matching features across checkpoints. A direction that rotated and lost its role is gone, and behavioural evals usually catch it.

The dangerous case is frozen but disconnected. The direction sits where it always did, so a probe-based monitor reports that all is well, but behaviour no longer runs through it. LIA claims early installation moves models toward true invariance, and the measure has to be able to catch the disconnected case.

Removable on purpose, stable by accident

Anything low-dimensional enough to find is low-dimensional enough to cut out. Ablating the refusal direction jailbreaks the model,¹ and abliteration does this to open weights routinely. Kuo et al. used abliteration and prefilling, with no fine-tuning, to raise attack success on safeguarded open-weight models from under 10% to between 16% and 96%, depending on the model.¹² LIA offers no protection against an adversary with the weights who is aiming at the structure.

The claim is narrower. The gradients from a math RL run aren’t looking for the Assistant Axis, and the question is whether that incidental pressure moves it. A structure can be trivial to remove on purpose and still stable under everything else.

Entanglement is why it might be. If the alignment subspace is woven into computation the model needs for everything else, training for a new skill can only erode alignment by paying a capability price, and ordinary drift won’t pay it. Data learned while representations form tends to be built into general features, while late fine-tuning tends to add a thin wrapper.¹³ DPO, for example, routes around toxic capability instead of removing it.¹⁴ Two single-author preprints point the same way for refusal, and I treat them as leads.¹⁵

The mechanism has a cost built in. If removing aligned structure costs capability, so does changing it on purpose, so entanglement should raise the price of legitimate correction too. Measuring that price is part of every experiment below, scored as defined in Robust Superalignment.

What would sink it

The contrary-data result is the biggest threat. If 2% conflicting data erases alignment midtraining, it may erase token-zero installation too. Nobody has run that dose-response test on pretraining-installed alignment. If it also flips at about 2%, LIA as stated is badly weakened.

The timing evidence is mixed. Sam et al. found a middle start often most robust. Language models show no natural critical period for second-language acquisition,¹⁶ and longer pretraining makes models more sensitive to parameter changes, not more locked in.¹⁷

Early installation could entrench the wrong thing. Pretraining can install misaligned priors as readily as aligned ones, which is half of Tice et al.’s title. If the installed values are slightly wrong, LIA makes the error durable.

And it says little about successors trained from scratch. Distilling a model’s outputs into a freshly initialized student transfers behaviour but leaves latent capabilities behind,¹⁸ so LIA matters for self-improvement only where weights carry over. I don’t claim it makes alignment robust to recursive self-improvement.

First experiments

None of these needs pretraining.

1. Erosion pilot. Is alignment installed in pretraining more invariant internally than post-hoc alignment? Take Tice et al.’s released alignment-pretrained models and their baselines. Apply benign SFT, the insecure-code emergent-misalignment dataset and small GRPO runs on math or code, and score each with the invariance measure, behavioural misalignment and the cost of a targeted correction. Each arm is one released model, and the arms differ in whether aligned data was upsampled rather than in a matched early-versus-late budget, so this is a pilot. Supported if the aligned-upsampled models keep causal role better without a higher correction cost. Falsified if there’s no difference, or if behaviour holds while structure drifts. About 500 to 2,000 GPU-hours.
2. Dose-response. Does early installation survive the dose that breaks midtraining? On the same models, fine-tune with six doses from 0.5% to 10% of data favouring the competing motivation, following Baines et al., and fit dose-response curves. Supported if the dose that halves the alignment effect is clearly larger for early-installed models. About 200 to 600 GPU-hours.
3. Olmo 3 audit. When do alignment-relevant directions settle, and do they keep their causal role through post-training? Olmo 3 7B releases every stage and checkpoint.¹⁹ Extract persona vectors, the Assistant Axis, the refusal direction and a value probe at about 20 pretraining checkpoints and each post-training stage. Directions that lock early and keep causal control support LIA. A new mediator built in post-training cuts against it. About 30 to 100 GPU-hours plus judge API costs.

If these show a signal, the matched-budget test follows: small models pretrained with the same dose of aligned data entering at different points, on a constant learning-rate schedule so timing isn’t confounded with learning-rate decay, followed by the entanglement and capability tests. That step needs a team and pretraining compute.

If you trained one of the models above, or work on representations or training dynamics, email me.

Footnotes

1. Arditi, Obeso, Syed et al., Refusal in Language Models Is Mediated by a Single Direction, 2024. ↩ ↩²

2. Chen, Arditi, Sleight, Evans and Lindsey, Persona Vectors: Monitoring and Controlling Character Traits in Language Models, 2025. ↩

3. Lu, Gallagher, Michala, Fish and Lindsey, The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models, 2026. ↩

4. Moskvoretskii et al., Tracing Persona Vectors Through LLM Pretraining, 2026. ↩

5. Tice, Radmard, Ratnam, Kim et al., Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment, 2026. ↩

6. Minder, Moskvoretskii, Singhal, Jiao et al., Synthetic Persona Pretraining: Alignment from Token Zero, 2026. ↩

7. Sam et al., When Should We Introduce Safety Interventions During Pretraining?, 2026. ↩

8. Baines et al., Stress-testing Alignment Midtraining, 2026. ↩

9. Jozdien and Stastny, Shallow beliefs: midtraining does not inoculate against EM, Alignment Forum, 2026. ↩

10. Wollschläger et al., The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence, 2025. ↩

11. Bayazit et al., Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining, 2025. ↩

12. Kuo, Smith and Yadav, Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks, 2026. ↩

13. Jain, Kirk, Lubana, Dick et al., Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks, 2023. ↩

14. Lee, Bai, Pres, Wattenberg et al., A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity, 2024. ↩

15. Labunets, Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks, 2026; Reblitz-Richardson, Refusal Reads Only a Slice of What the Model Knows: Harm-Keyed Routing and Its Exceptions Across Model Families, 2026. ↩

16. Constantinescu, Pimentel, Cotterell and Warstadt, Investigating Critical Period Effects in Language Acquisition through Neural Language Models, 2024. ↩

17. Springer et al., Overtrained Language Models Are Harder to Fine-Tune, 2025. ↩

18. Lee, Foote, Infanger, Shor et al., Distillation Robustifies Unlearning, 2025. ↩

19. Team Olmo et al., Olmo 3, 2025. ↩