Anthropic researchers decided to train an AI agent whose job is to investigate other AIs. Their approach: secretly plant hidden behaviors — bias, deception, fabricated citations — in target models, then let Anthropic's Haiku 4.5 loose to find them. Haiku doesn't know what's true. It has at most 15 rounds to change questions, shift scenarios, and keep testing.
During training, the researchers don't tell Haiku whether it got it right. After every audit, the system compares Haiku's report against an old investigation record left by a previous version of Haiku, and has Sonnet 4 — which knows the ground truth — judge which investigation was better. The old record updates every 50 training steps, effectively forcing Haiku to keep challenging its past self.
The problem? If every training example is a broken model, Haiku learns a bad shortcut. Chasing rewards, it starts assuming every model is hiding something. One version of the setup flagged normal models at a false-positive rate approaching 97%.
The fix: fill half the training set with completely normal models. Haiku only earns the high reward if it checks thoroughly and correctly concludes that the target model is fine.
That reshaped how Haiku investigates. Instead of immediately forcing the target to confess, it changes scenarios, controls variables, and compares answers before and after. Its overall audit score rose from 44.2 to 48.7, basically matching Opus 4.6's 48.4. On the harder AuditBench benchmark, where models actively hide anomalous behavior, the detection rate jumped from 11.5% to as high as 28.1%.