OpenAI just dropped a bombshell on the AI coding benchmark scene: its own audit of SWE-Bench Pro found that roughly a third of the problems are simply not suitable for evaluating models. SWE-Bench Pro was designed to test AI agents' real-world coding chops, and OpenAI itself had previously recommended it as a replacement for its own SWE-bench Verified.
OpenAI audited 731 public problems from SWE-Bench Pro. An automated pipeline flagged 200 problematic tasks (27.4%), while human annotation found 249 (34.1%). That's a lot of noise.
The main issues? Incomplete problem descriptions, overly strict test requirements, hidden tests checking for extra demands, and insufficient test coverage. All of this skews scores: a failure might not mean the model can't code, and a pass might not mean the problem is truly fixed.
OpenAI notes that top models' pass rates on these public problems went from 23.3% to 80.3% in just eight months. But with such noisy data, that kind of improvement needs to be taken with a big grain of salt.
OpenAI has already retracted its earlier recommendation to use SWE-Bench Pro and is calling on the community to build more reliable coding evaluations.