Scalable oversight for AI-written research

Oversight evidence is strongest where a judge can check something specific, or holds information the model lacks. It’s thinnest where the task is fuzzy and the model knows more than the reviewer. AI-written alignment research sits in that second place. Agents will soon produce more results than people can read closely, much of alignment research has no answer key, and a persuasive wrong result is worse than none when it feeds a decision about deploying or training a model.

This post is narrower than the rest of the agenda. It asks one question: what helps a reviewer check claims in a research write-up they couldn’t check alone?

The hypothesis

Many of the errors in AI-written alignment research will be inferential: claims that outrun their evidence, missing baselines, proxy metrics treated as the goal. For those, a single adversarial critique, from a model better than the reviewer at spotting flaws and then checked by the reviewer, should recover most of what debate gains, at a fraction of the cost. This extends Elasky et al.’s critique-versus-debate result from verifiable code and logic to research review. It only counts if critique beats the simplest alternative, a reviewer who defers to the model, on flaws with known answers.

It’s false if critique doesn’t beat deference for non-expert reviewers. It’s weakened if debate clearly beats a single critique on this task, or if most real errors turn out to be code and data faults rather than inferential ones.

What the evidence says

Debate, in which two models argue opposite answers before a weaker judge, helps under specific conditions.¹ On QuALITY, where debaters can read a hidden passage and the judge can’t, debate raised judge accuracy to 76% for model judges and 88% for humans, against naive baselines of 48% and 60%.² At scale, Kenton et al. found debate beat a direct judge clearly only when debaters held information the judge lacked.³

Two newer results shape the design. On verifiable tasks, Elasky et al. found debate’s advantage depends on the critic telling sound answers from flawed ones better than the judge does, and on the judge treating critiques as claims to check. Where that held, a single independent critique recovered “the bulk of debate’s benefit”.⁴ Recchia et al. ran simple protocols with human evaluators and found no overall advantage; people grew more confident in wrong model answers after doing their own research.⁵ So plain deference has to be reported every time.

The sharpest warning comes from Wen et al. RLHF made models better at convincing time-limited humans without making them more correct, and human false-positive rates rose 24.1% on QuALITY and 18.3% on APPS.⁶ A critique that persuades is not the same as one that is right.

Sabotage benchmarks plant flaws in code and data.⁷ Bowkis et al. locate the harder failure in judgment: systematic errors, and correct results aggregated into overconfident safety assessments.⁸ I haven’t found a testbed with known answers for that layer. FindTheFlaws, a set of reasoning traces with expert-annotated flaws, is the closest, and I’d build on it.⁹

First experiments

1. What fails in practice? Alignment researchers classify the errors in a sample of agent-written alignment studies as inferential, code or data faults. If inferential errors are rare, the hypothesis loses its motivation, which is why this runs first. About two weeks of expert time.
2. Critique against debate, models only. Build a corpus of results sections from recent alignment papers, published after the models’ training cutoffs, with expert-planted flaws (overclaiming, missing baselines, seed cherry-picking, evaluation leakage, wrong aggregation) and certified clean controls. Weak open models (1B to 8B) judge; frontier models critique and debate. Compare a direct judge, deference, consultancy, single critique and two-round debate on every item. Critique matching debate within a 5-point margin set before the experiment, with gains that depend on the critic out-classifying the judge, supports the hypothesis. About $3,000 to$6,000 in API calls plus expert annotation.
3. Human sandwiching. Does critique help ML-literate non-experts beyond deferring to the model?¹⁰ First confirm the model beats unaided non-experts on these items and experts beat the model. Then compare no aid, the model’s answer alone and critique. Critique beating deference on flawed items without raising false alarms on clean ones supports it. Rising confidence in wrong verdicts, as Recchia found, falsifies it. About $10,000 to$20,000 for participants.

Later, the same corpus tests whether reviewers from a different model family catch flaws that same-family reviewers miss, which is shared with Automating alignment research, and a red team that optimizes flaw insertion against the best protocol.

If you know the debate and sandwiching literature well, I’d value your view, especially if you think I’ve misread a result. Email me.

Footnotes

1. Irving, Christiano and Amodei, AI safety via debate, 2018. ↩

2. Khan, Hughes, Valentine, Ruis et al., Debating with More Persuasive LLMs Leads to More Truthful Answers, 2024. ↩

3. Kenton, Siegel, Kramár, Brown-Cohen et al., On scalable oversight with weak LLMs judging strong LLMs, 2024. ↩

4. Elasky, Nakasako and Goyal, Debate Helps Weak Judges Reward Stronger Models, 2026. ↩

5. Recchia, Mangat, Nyachhyon, Sharma et al., Confirmation bias: A challenge for scalable oversight, 2025. ↩

6. Wen, Zhong, Khan, Perez et al., Language Models Learn to Mislead Humans via RLHF, 2024. ↩

7. Libon, Rank, Yeon, Schmotz et al., ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D, 2026. ↩

8. Bowkis, Buhl, Pfau and Irving, Automated alignment is harder than you think, 2026. ↩

9. Recchia, Mangat, Li and Krishnakumar, FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research, 2025. ↩

10. Bowman, Hyun, Perez, Chen et al., Measuring Progress on Scalable Oversight for Large Language Models, 2022. ↩