Anthropic researchers discovered that asking a model to critique itself is a recipe for overconfidence. So they tried something else: spinning up an independent judge.
https://twitter.com/RLanceMartin/status/2064397389189071163
In a machine learning training challenge, the team pitted Fable 5 against Opus 4.7. Instead of having Fable 5 self-critique, they used CMA's Outcomes feature to spawn a scoring agent in a separate context window. That agent acted as a judge, following a spec document with nine metrics.
The result? The loop design with an independent judge let Fable 5 improve its training pipeline by six times as much as Opus 4.7. Fable 5 wasn't afraid to try major architecture changes and stubbornly fixed issues even when quantization caused rollbacks. Opus 4.7, by contrast, got stuck tweaking scalar constants according to a template — a conservative rut.
In a separate experiment on cross-session online learning for database QA, the team used a shared mountable filesystem to develop a five-step memory method: record failures, diagnose causes, verify facts, extract general rules, and directly retrieve those rules.
The results showed a stark performance ladder. Sonnet 4.6 half-heartedly logged errors and made blind guesses, almost never checking history. Opus 4.7 built pattern references with uncertainty markers but only verified about 17% of them. Fable 5 completed the full memory chain, achieving 73% verification coverage and distilling diagnoses into general rules.
The takeaway: designing self-correction loops that respond to feedback, paired with autonomous memory management, is far more practical than crafting prompts to guide a large model.