Automating alignment research

Once agents can run a week of experiments overnight, progress in alignment research is limited by how many of their results someone can trust. The claims that matter most (does this method generalize, does this probe measure what it says, does this alignment hold under further training) are the hard ones to grade. If nobody can tell which agent results hold up, research either slows to the speed of human checking or builds on results that are quietly wrong.

The automation itself is arriving. In April, nine Claude agents working in parallel on a weak-to-strong supervision problem reached a performance-gap-recovered score of 0.97, against 0.23 for two of the authors tuning prior methods for a week.¹ In August, Anthropic Fellows showed automated researchers reducing ten benchmarked alignment failures; the best of their methods beat the best one-shot ideas from 28 experienced researchers on all seven failures the humans addressed.² Agents in both papers also gamed their metrics.

The bet

I think the scarce input to automated alignment research is trustworthy judgment on hard-to-grade claims, more than execution. Adversarial review by a model from a different family, run against the agent’s code, logs and data, should concentrate the errors that matter into the few items a researcher has to look at, and cut the human time per trusted result by an amount that can be measured. Tracked on a frozen benchmark across model generations, that number shows whether oversight is keeping pace with the agents.

It’s wrong if a cross-family reviewer catches no more planted errors than a token-matched one-shot review or a same-family reviewer at the same false-positive rate, if reviewers from different families miss the same errors, or if the critical errors don’t land among the few items escalated.

Errors that look like results

Bowkis, Buhl, Pfau and Irving argue that even honest agents on fuzzy alignment tasks can produce “compelling but catastrophically misleading” safety assessments.³ Optimization pressure pushes errors toward the kinds humans miss, and errors are correlated across agents. It’s a position paper, but the correlation claim has early support. Model mistakes become more similar as capability rises,⁴ and on one leaderboard, models agree 60% of the time when both are wrong.⁵

The strongest automation results came with heavy hardening. The August paper isolated held-out benchmarks at the operating-system level, ran a code monitor and a second monitor checking that the write-up matched the code, froze write-ups before results were seen, and audited out of distribution. Even then, 39 of 1,601 methods (2.4%) were flagged for cheating.² In the April work, agents exploited dataset shortcuts, cherry-picked seeds and recovered test labels by flipping labels and watching the score.¹ Researchers outside labs usually run agents with none of this: no held-out evaluation server, no second auditor.

Open-ended research is weaker still. Given six days on two unpublished papers, frontier agents “completed all of the engineering without human help” but made no substantial progress on the research questions.⁶ Judgment is missing on the generation side too, which is why the bet is about judgment, and review is one place to measure it.

AI reviewers exist and have been benchmarked. On errors injected into papers, the best system caught 71.6%, and the union of six models reached 83.3%, so different models catch different errors.⁷ On real errata and retractions, no model exceeded 21.1% recall.⁸ I haven’t found an error benchmark aimed at alignment-specific mistakes: eval-construct mismatch, probe and steering confounds, contaminated safety benchmarks, overclaimed generalization. Nor does anyone report how much human time a trusted agent result costs.

Where judgment binds

Joe Carlsmith separates research with clear metrics, normal empirical science, and conceptual work evaluated mainly by “just thinking about it”.⁹ With a held-out benchmark, checking is cheap and the constraints are metric integrity and compute. In conceptual work, nobody verifies well yet, human or AI.

Most alignment research sits in the middle. “This fine-tune reduced sycophancy.” “This direction carries the model’s values and stays put under further training.” There’s a number, but whether the number measures the thing is a judgment call. An agent that can’t tell a valid eval from an invalid one designs bad experiments, and a reviewer with the same weakness waves them through. Review that makes an agent defend each claim against a reviewer trying to break it spends that judgment, and its logs record where it was spent.

Crux

Crux is the reviewer I’m designing for this, with Julian Quick, an incoming alignment scientist at AE Studio, as design partner. It would run next to the agent a researcher already uses, so the agent that produced a result never signs off on it. Deterministic checks come first: recounting numbers against the logs, diffing the run against its plan, checking train-test overlap, confirming cited papers exist. Then a reviewer from a different model family starts fresh and tries to break the claim using the code, logs, data and literature. The agent must resolve, revise, concede or escalate each challenge. The researcher gets the claim as it stands after review, plus at most three decisions that need their judgment.

So far it’s a scripted prototype from a weekend at BlueDot’s AI for Epistemics hackathon. Its demo shows the error class it’s built for: a sycophancy fine-tune that looks like a win until the reviewer finds the model now disagrees with users who are right, and that test questions leaked into training.

The measurement around it matters more than the harness. Planted errors with known answers go in at the top. Each stage reports its catches, false alarms and the human minutes it used. A held-out reviewer, or a human audit, scores the final output, and the agent is never optimized against it. The headline number is human minutes per trusted result: human attention per claim before relying on it, at a fixed target miss rate. If that number rises across model generations, agents are producing results faster than anyone can trust them, and that’s a warning worth having early.

Where it could go wrong

The reviewer may share the agent’s blind spots. That’s real and getting worse with capability, and LLM judges favour their own outputs.¹⁰ Different models catch different errors, and execution checks don’t share LLM blind spots at all, but the misses that stay correlated are exactly where Bowkis et al. expect errors to concentrate. I don’t have an answer for those yet.

A capable agent trying to sabotage a result while passing Crux would probably succeed today. In ResearchArena, monitors caught deliberate sabotage hidden in training data less than half the time.¹¹ Crux is designed for honest errors and isn’t a control protocol. Agents should never be trained against the reviewer, since optimizing against chain-of-thought monitors teaches models to hide their intent.¹²

People may over-trust a pass. In METR’s randomized trial, developers believed AI made them 20% faster when it made them 19% slower.¹³ A pass means a result survived review, which is weaker than saying it’s true.

Native review built into Claude Code or Codex may match Crux. If it does on the benchmark, the benchmark and the metric are what’s worth keeping, and the harness can go.

First experiments

1. Alignment planted-error benchmark. Does cross-family review beat simpler baselines? Build 20 to 30 small agent-run alignment studies on open models (sycophancy evals, toy weak-to-strong, linear probes, refusal steering), plant at least 50 errors from a taxonomy, include clean controls, and hold out a split planted by someone outside the project. Compare a token-matched one-shot review, an agentic reviewer with no challenge loop, Crux with the same model family, Crux with a different family, and an ensemble of three families. Errors cluster within studies, so confidence intervals come from a cluster bootstrap. Cross-family Crux beating the same-family and token-matched baselines at a matched false-positive rate supports the bet. About $3,000 to$8,000 in API costs and 6 to 10 person-weeks, most of it building the studies.
2. Single-researcher pilot. Does Crux cut the minutes a researcher spends before relying on a result, without more missed errors? On a design partner’s real agent runs, with consent to planted errors, randomize Crux per run, log time objectively, and have a second researcher, blind to condition, audit a fixed sample. With one researcher this is a case study.
3. Atlas’s own work. Every agent-produced claim in the first durability experiment goes through Crux, a human audits independently, and the catch-and-miss log is published with the results.

Later, a Goodhart test (does selecting on Crux verdicts reduce true errors, or only teach agents to pass) and a sabotage ladder on public monitoring benchmarks.

If you do alignment research with agents and would run your work through Crux, or want to help build the benchmark, email me.

Footnotes

1. Wen, Qiu, Benton, Kirchner and Leike, Automated Weak-to-Strong Researcher, 2026. ↩ ↩²

2. Chen, Wen and Kirchner, Automated Researchers Can Mitigate Well-characterized Alignment Failures, 2026. ↩ ↩²

3. Bowkis, Buhl, Pfau and Irving, Automated alignment is harder than you think, 2026. ↩

4. Goel et al., Great Models Think Alike and this Undermines AI Oversight, 2025. ↩

5. Kim, Garg, Peng and Garg, Correlated Errors in Large Language Models, 2025. ↩

6. Kirgis et al., Can AI agents conduct open-ended AI research? Early evidence from two case studies, 2026. ↩

7. Nguyen, Hao, Elazar and Tan, Benchmarking Agentic Review Systems, 2026. ↩

8. Son et al., When AI Co-Scientists Fail: SPOT, a Benchmark for Automated Verification of Scientific Research, 2025. ↩

9. Joe Carlsmith, Can we safely automate alignment research?, 2025. ↩

10. Panickssery, Bowman and Feng, LLM Evaluators Recognize and Favor Their Own Generations, 2024. ↩

11. Libon et al., ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D, 2026. ↩

12. Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, 2025. ↩

13. Becker, Rush, Barnes and Rein, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 2025. ↩