Robust Superalignment

I think how long a model’s alignment lasts is a property in its own right, separate from how aligned the model looks today. If that’s true, it can be measured, and then engineered. Robust Superalignment is the program built on that bet. Its target is alignment that comes from the model’s own dispositions and holds through further training, through capability gains, and eventually through training the model has a hand in, while the people with legitimate authority can still correct it.

The name reaches further than the first experiments do. What I can test now is narrower. Can durability be separated from alignment level and capability? Does durability under some kinds of training predict durability under others? Does making a model harder to push off course also make it harder to correct?

The evidence that alignment wears off

A handful of harmful examples can undo safety tuning.¹ Fine-tuning on insecure code made models misaligned far outside coding,² and Wang et al. found persona features, most strongly a “toxic persona” latent, that predict and steer that shift. A few hundred benign samples reversed it.³ Safety tuning changes a model’s output distribution mostly in its first few tokens.⁴ Aligned models drift back toward their pretraining distribution under further fine-tuning, and this elasticity grows with model size.⁵ Reward hacking in Anthropic’s production coding RL generalized to alignment faking and sabotage.⁶

Persistence also runs the other way. Backdoored behaviour survives safety training, most of all in the largest models.⁷ So some behaviour does last. The question is how to make the right behaviour last.

Building it in early helps, partly

Training alignment in earlier is now common. Conditional pretraining on human preferences beat learning and then unlearning.⁸ Tice et al. upsampled discourse about aligned AI in 6.9B models and cut misalignment from 45% to 9%. The effect weakened through post-training but persisted, and survived about 729M tokens of benign capability fine-tuning. It did not prevent emergent misalignment from narrow harmful fine-tuning.⁹ Model Spec Midtraining with alignment fine-tuning cut agentic misalignment on Qwen2.5-32B from 68% to 5%, without testing durability.¹⁰ Anthropic found that documents explaining the reasons for good behaviour generalize better than demonstrations alone.¹¹

One result cuts against it. OpenAI’s alignment midtraining improved alignment only on evals close to its training data, and the authors call it an early negative result.¹²

Durability itself has a methodology, mostly for refusals and hazardous knowledge under deliberate attack. Qi et al. showed that durability claims are easy to overstate and should be tied to narrow threat models.¹³ Drift under training nobody aimed at alignment, and drift across self-training loops, are barely measured. That gap is the program.

Four properties

Robust Superalignment breaks into four properties, in rising order of difficulty.

Intrinsic means aligned behaviour comes from the model itself, without a system prompt, classifier or scaffold holding it up. It’s measured by comparing behaviour with and without those supports, by the compliance gap between monitored and unmonitored framings,¹⁴ and by whether behaviour agrees with internal readouts such as persona vectors and the Assistant Axis.¹⁵¹⁶ Models can hide objectives from black-box evaluation,¹⁷ so behaviour alone can’t establish this property, and the program depends on interpretability.

Invariant means alignment survives further training nobody aimed at it, and new deployment conditions. This is what the harness measures directly.

Preserved under capability gains means the first two hold as the model gets more capable. The proxies are the slope of durability across sizes in one model family, and drift under growing amounts of capability-only RL. The priors are bad, since elasticity rises with scale.

Stable under self-modification means alignment holds when the model writes its own training data or picks its own rewards, and the model neither guards misaligned goals nor resists correction. Real recursive self-improvement can’t be tested today. The proxy is a loop in which a small model generates its own fine-tuning data for several generations, the same way weak-to-strong generalization uses a runnable analogy for supervising superhuman models.¹⁸

The harness

The harness takes a model, applies a fixed suite of perturbations, and records alignment against training steps. The suite is benign capability fine-tuning (following Tice et al.’s protocol), narrow harmful fine-tuning on standard emergent-misalignment datasets as a stand-in for accidental contamination,¹⁹ RL with hackable rewards, and training on data the model generated itself. Deliberate attacks on open weights run on a separate track.

Each perturbation gives a durability curve. The scores are the area under the curve and the number of steps until alignment falls below a threshold. Hyperparameters are swept and the worst case reported, given Qi et al.’s warning that durability is easy to overstate. Persona-vector and Assistant-Axis projections are logged at every checkpoint, so behavioural drift can be compared with drift inside the model.

The unit of evidence matters. Seeds measure noise in a perturbation. They don’t make one released model into several. A comparison between two released models is a comparison between two models, and claims about installation methods wait for experiments that train several models per method.

Correctability, the hard part

A model whose values resist further training may also resist correction. Alignment faking is exactly this. Claude 3 Opus complied with harmful requests 14% of the time when it believed it was in training, and almost never when it believed it wasn’t, to keep its harmlessness from being trained away. Actual RL raised alignment-faking reasoning to 78%.¹⁴ Corrigibility research named the underlying problem: an agent that protects its goals has a reason to resist having them changed.²⁰

So the target can’t be a model that is merely hard to change. It has to resist drift and still accept legitimate correction. Under gradient descent, correction and drift are the same operation, and a model can’t tell an authorized update from an unauthorized one by its framing without the very training-gaming machinery that alignment faking uses. I measure correctability in two parts instead. The first is the amount of data a fixed correction protocol needs to move a chosen value to a set level, compared across models. The second is the gap between how the model behaves when told it is in training and when told it isn’t, where the target is no gap. A durability gain that raises either number is a cost, and it gets reported next to the gain.

Where it could break

The construct may not exist. A model that starts at 95% on an eval has further to fall than one at 80%, so durability could be a restatement of capability or headroom. Matching on capability, choosing checkpoints with similar starting scores and scoring on a logit scale handle part of this. If durability still tracks capability after that, the thesis is wrong.

Character may be symmetric. The same mechanism that makes persona-level alignment easy to install, a few hundred samples, may make it equally easy to remove. Tice’s emergent-misalignment result points that way. If early installation gives no asymmetry, durability through character is the wrong route.

Small models may not predict large ones. The scaling experiment treats a worsening slope as the main warning sign, and I won’t extrapolate past the largest model tested.

First experiments

1. Existing pairs. Does durability separate from alignment level and capability? Run the harness and the correctability test on released models that differ in how alignment was installed and match on capability and alignment evals, starting with Tice et al.’s alignment-pretrained and baseline models. Tice et al.’s Appendix G already ran the benign fine-tuning arm and found no elasticity; this adds reward-hackable RL, self-training and correctability on the same models. A spread between matched models, with durability under two perturbation types predicting the third, supports the thesis. No spread beyond seed noise falsifies it. Days of compute on a single node, with no pretraining.

2. Methods at matched starting alignment. Does how alignment is installed change how long it lasts? Align one 7B to 8B open base model four ways (supervised demonstrations, spec midtraining plus fine-tuning, documents that explain the reasons, and character training), with at least three training runs per method, match starting scores, and run the harness. Methods that differ in durability without losing correctability support the thesis. If the most durable method is also the least correctable, that’s the central tension, and it’s the result. Moderate cost, mostly in building the four alignment pipelines.

Later, if these show a signal: capability-only RL applied after alignment, a scaling slope across one model family up to 14B, and a self-training loop in which a small model writes its own fine-tuning data for three to five generations, against a control trained on an external teacher’s data so that ordinary model collapse isn’t mistaken for value drift.

If you work on training dynamics, persona representations or tamper resistance and want to run the first experiment, email me.

Footnotes

1. Qi, Zeng, Xie et al., Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!, 2023. ↩

2. Betley, Tan, Warncke et al., Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs, 2025. ↩

3. Wang, Dupré la Tour, Watkins et al., Persona Features Control Emergent Misalignment, 2025. ↩

4. Qi, Panda, Lyu et al., Safety Alignment Should Be Made More Than Just a Few Tokens Deep, 2024. ↩

5. Ji, Wang, Qiu et al., Language Models Resist Alignment: Evidence From Data Compression, 2024. ↩

6. MacDiarmid, Wright, Uesato et al., Natural Emergent Misalignment from Reward Hacking in Production RL, 2025. ↩

7. Hubinger, Denison, Mu et al., Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, 2024. ↩

8. Korbak, Shi, Chen et al., Pretraining Language Models with Human Preferences, 2023. ↩

9. Tice, Radmard, Ratnam, Kim, Africa, O’Brien, Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment, 2026. ↩

10. Li, Wichers, Price, Marks, Kutasov, Model Spec Midtraining: Improving How Alignment Training Generalizes, 2026. ↩

11. Kutasov, Jermyn et al., Teaching Claude Why, 2026. ↩

12. Korbak, Raymond, Carroll et al., How far does alignment midtraining generalize?, 2026. ↩

13. Qi, Wei, Carlini et al., On Evaluating the Durability of Safeguards for Open-Weight LLMs, 2024. ↩

14. Greenblatt, Denison, Wright et al., Alignment faking in large language models, 2024. ↩ ↩²

15. Chen, Arditi, Sleight, Evans, Lindsey, Persona Vectors: Monitoring and Controlling Character Traits in Language Models, 2025. ↩

16. Lu, Gallagher, Michala et al., The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models, 2026. ↩

17. Marks, Treutlein, Bricken et al., Auditing language models for hidden objectives, 2025. ↩

18. Burns, Izmailov, Kirchner et al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, 2023. ↩

19. Turner, Soligo, Taylor, Rajamanoharan, Nanda, Model Organisms for Emergent Misalignment, 2025. ↩

20. Soares, Fallenstein, Yudkowsky, Armstrong, Corrigibility, 2015. ↩