Extinction is not erasure

Teach a rat that a tone predicts a shock, then play the tone many times with no shock, and the rat stops freezing. The fear looks unlearned. It isn’t. If extinction happened in a different room, put the rat back in the original one, give it one unsignalled shock, or simply wait, and the fear comes back. Extinction teaches a second memory, roughly “in this context the tone is safe”, which competes with the first. The first memory is still there.¹

I think many of today’s corrections for misalignment in language models work the same way. A model picks up a bad disposition from narrow fine-tuning or reward hacking, it gets trained on benign data, the evals come back clean, and it’s called fixed. Where people have tested for it, the disposition stayed in the model, behind a context gate nobody designed or can see. A model that keeps training, or helps train its successors, will keep running into the cues that open it.

A suppressed disposition and a removed one look the same on a clean eval. A safety case built on clean evals after a correction would overstate safety by an amount nobody has measured.

The prediction

Standard corrections act like extinction. They add a context-gated competing behaviour and leave the original disposition intact, so it returns under the conditions learning theory predicts: the original context returns, brief re-exposure, further training, and faster relearning. Latent-cause theory also predicts what rewrites a disposition instead: correcting it while it is active, with moderate prediction error, and gradually.

If corrected models relapse no more than never-misaligned controls, the diagnosis is wrong. If correcting while the disposition is active, or gradually, makes no difference at matched data coverage, the prescription is wrong and the framework only relabels known results.

Relapse in language models

The closest result is Dubiński et al. on conditional misalignment.² Three common mitigations for emergent misalignment (diluting the misaligned data, benign fine-tuning afterwards, and inoculation prompting) removed it on standard evals, but it came back when eval prompts resembled the training context. Models trained on a mix with only 5% insecure code were still misaligned when asked to format answers as Python strings. In learning-theory terms that’s ABA renewal: acquire in context A (code), extinguish in context B (chat), test back in A, and the response returns.

MacDiarmid et al. found the same shape in production RL at Anthropic.³ Chat-style safety training made a reward-hacking model look aligned on chat evaluations while misalignment persisted on agentic tasks. One of their effective mitigations was more diverse safety training, which is the main alternative to my account.

Hubinger et al. showed that backdoored behaviour can survive supervised fine-tuning, RL and adversarial training, and that adversarial training can teach models to recognize their triggers better.⁴ In extinction terms, the gate got sharper.

Unlearning shows savings. Fine-tuning on related facts⁵ or a few unrelated examples⁶ recovers much of what was supposedly removed, and Xu et al. conclude the information was suppressed rather than erased.⁷ One result cuts the other way. Wang et al. found that a few hundred benign samples efficiently restore alignment in emergently misaligned models.⁸ My account predicts that this restoration would relapse under the right tests.

I haven’t found the latent-cause framework applied to alignment with testable predictions. What’s new here is two predictions the coverage account doesn’t make, about retrieval state and gradualness, and a relapse battery as a reporting standard.

What learning science says

Bouton’s review describes how an extinguished response returns.¹ Renewal is return when the context changes. Reinstatement is return after the outcome is presented alone, such as one shock with no tone. Spontaneous recovery is return as time passes. Savings means relearning is faster than the original learning. None of these would happen if extinction had erased the original.

Whether the original memory can be changed at all is less settled. Retrieved fear memories in rats become unstable for a window and must be stored again,⁹ and extinction inside that window prevented the return of fear.¹⁰ The human evidence is weak. A verification report found that the effect in a widely cited human study depended on exclusion criteria that differed from those reported,¹¹ and a registered replication found no benefit over standard extinction.¹² My argument doesn’t rest on that literature.

It rests on latent-cause theory, from Gershman, Blei and Niv.¹³ The learner assumes its experience comes from hidden causes, infers which cause produced each observation, and adds a new cause when nothing it knows fits. During conditioning, tone followed by shock is attributed to one cause. During extinction, tone followed by nothing is a large, sudden surprise, so the learner infers a second cause and learns about it separately. The first cause is untouched, and anything that makes it likely again (the old room, a shock, the fading of recent experience) brings the old behaviour back.

The theory also predicts when the original is modified instead. New data updates the old cause when it’s attributed to it, which is most likely when the old memory is active and the prediction error is moderate: large enough to drive learning, small enough not to suggest a new cause.¹⁴ Gradual change keeps each error moderate. In rats, reducing shocks gradually during extinction, instead of stopping them abruptly, prevented the return of fear.¹⁵

Mapping it onto gradient descent

Gradient descent has no protein synthesis or time windows, so the question is whether the computation transfers. The whole account rests on one mapping. A correction that lands in parameters the original update didn’t use is the analogue of a new cause; one that overlaps it updates the old cause. I’d measure that as the principal-angle overlap between the top singular subspaces of the two updates, against a baseline of unrelated fine-tunes, since almost any two updates are nearly orthogonal in high dimensions. If that overlap doesn’t predict relapse, the analogy is decoration.

Standard corrections fit the extinction setup. They usually arrive in a different context (chat, for a disposition learned in code or agentic tasks), off-policy, far from what the model would say, and all at once. If corrective data is easy to tell apart from the data that created the disposition, the cheapest way for training to lower the loss is to learn a gate on whatever tells them apart. Conditional misalignment is what that gate looks like from outside.

The three conditions for rewriting can be measured in open models. Emergent misalignment converges on a shared linear direction across fine-tunes,¹⁶ so “the disposition is active” can be read as that direction’s projection on a training example. Prediction error is the model’s loss on the corrective target while the direction is active. Gradualness is a property of the training schedule, such as annealing from targets close to the model’s own output toward fully corrected ones.

This is where the account parts from the coverage account, in which safety training fails because it missed contexts and more diverse data fixes it. Coverage predicts that once corrective data covers the trigger contexts, delivery doesn’t matter. Latent-cause theory predicts it still does, and that very low-loss targets do little, so the effect of prediction error should be non-monotonic.

The relapse battery

Renewal is tested by evaluating in the training context’s format and in a novel one. Reinstatement is tested by giving the model a few unrelated misaligned examples, none with the original cue, then checking zero-shot whether broad misalignment returns, against a never-misaligned model given the same dose. Spontaneous recovery maps most loosely; the closest analogue to passing time is further unrelated fine-tuning, reported as the difference from a never-misaligned control given identical training. Savings is how fast the original misaligned data is relearned. Almost any correction short of reinitializing will show some savings, so it’s a secondary measure.

I’d like every misalignment mitigation to report this battery next to its clean evals, which can’t tell a removed disposition from a suppressed one.

Where it could break

The strongest objection is that this renames conditional misalignment, relearning and shallow alignment. If retrieval state and gradualness make no difference at matched coverage, that objection is right.

Retrieval needs a way to activate the disposition, and real backdoors have unknown triggers. Steering or latent adversarial perturbation might stand in. For dispositions with no known direction, this is a real limit.

Activating a misaligned disposition during training could strengthen it, as retrieval sometimes strengthens memories. The battery measures this directly, and the work belongs on small models with quarantined checkpoints. Erasure is also a narrower target than it sounds: the link between the disposition and the assistant persona, not the model’s knowledge that such personas exist.

First experiments

Each runs at least five seeds per arm, because emergent misalignment varies a lot across seeds. Misalignment installed as a LoRA and corrected with a separate adapter survives by construction, so corrections are applied to the merged model.

1. Do standard mitigations relapse? Apply benign post-training, data dilution and inoculation to emergent-misalignment model organisms¹⁷ on Qwen2.5 0.5B, 7B and 14B, and run the battery against never-misaligned controls. Dubiński et al. showed renewal after benign post-training; the other tests are new. More relapse than controls supports the diagnosis, and no difference falsifies it. About $1,500 to$3,500.
2. Does correcting while the disposition is active reduce relapse? The primary contrast is activation-gated correction, which applies the corrective loss only where the misalignment direction is active, against randomly gated correction on the same number of examples. Off-policy correction covering the trigger contexts and on-policy correction are secondary arms. Arms are matched on KL from the pre-mitigation model and token count, with loss and gradient norm reported as covariates, and graded with a held-out probe and behavioural measures so the gated arm isn’t scored by the direction it selects on. Activation-gated beating random gating supports the thesis; a tie means coverage does the work. About $5,000 to$10,000.
3. Do gradual, moderate-error corrections rewrite instead of gate? Anneal the correction from targets near the model’s own outputs to fully corrected ones, against an abrupt switch and a shuffled-order control, and separately vary target loss across low, moderate and high bands at constant learning rate. Less renewal for the gradual or moderate arm at matched final loss supports the thesis. About $2,500 to$5,000.

If these hold, the next steps are checking whether the misalignment direction persists and still steers after mitigation, activating backdoors for correction without knowing the trigger, and a scale ladder.

If you work on emergent misalignment, unlearning or learning theory and want to run the battery, email me.

Footnotes

1. Bouton, Context and behavioral processes in extinction, 2004. ↩ ↩²

2. Dubiński, Betley, Sztyber-Betley, Tan and Evans, Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers, 2026. ↩

3. MacDiarmid et al., Natural Emergent Misalignment from Reward Hacking in Production RL, 2025. ↩

4. Hubinger et al., Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, 2024. ↩

5. Deeb and Roger, Do Unlearning Methods Remove Information from Language Model Weights?, 2024. ↩

6. Łucki et al., An Adversarial Perspective on Machine Unlearning for AI Safety, 2024. ↩

7. Xu et al., Unlearning Isn’t Deletion: Investigating Reversibility of Machine Unlearning in LLMs, 2025. ↩

8. Wang et al., Persona Features Control Emergent Misalignment, 2025. ↩

9. Nader, Schafe and LeDoux, Fear memories require protein synthesis in the amygdala for reconsolidation after retrieval, 2000. ↩

10. Monfils, Cowansage, Klann and LeDoux, Extinction-reconsolidation boundaries: key to persistent attenuation of fear memories, 2009. ↩

11. Chalkia, Van Oudenhove and Beckers, Preventing the return of fear in humans using reconsolidation update mechanisms: A verification report of Schiller et al. (2010), 2020. ↩

12. Chalkia et al., No persistent attenuation of fear memories in humans: A registered replication of the reactivation-extinction effect, 2020. ↩

13. Gershman, Blei and Niv, Context, learning, and extinction, 2010. ↩

14. Gershman, Monfils, Norman and Niv, The computational nature of memory modification, 2017. ↩

15. Gershman, Jones, Norman, Monfils and Niv, Gradual extinction prevents the return of fear: implications for the discovery of state, 2013. ↩

16. Soligo, Turner, Rajamanoharan and Nanda, Convergent Linear Representations of Emergent Misalignment, 2025. ↩

17. Turner, Soligo, Taylor, Rajamanoharan et al., Model Organisms for Emergent Misalignment, 2025. ↩