Stanford, Oxford, and the Allen Institute for AI have teamed up to create CUSP, a time-sensitive benchmark that tests how well large language models can predict scientific progress. They put GPT-5.4, Claude Sonnet 4.5, and DeepSeek R1 through the paces. The verdict? LLMs are great at understanding established research paths, but when it comes to guessing whether a new discovery will actually happen, they're barely better than a coin flip. And their timeline predictions? Systematically behind the curve.
Traditional AI evaluations are easy to game — models can just regurgitate papers they've already seen. To get at true predictive ability, CUSP introduces a knowledge cutoff: it only gives models information available up to a certain date, then asks them to forecast what happens next. The team curated 4,760 scientific milestones from journals like Nature and Science, generating 17,429 specific test tasks. They also ran control experiments with pre-cutoff web searches to separate knowledge gaps from prediction gaps.
So how did the models do? On mechanistic reasoning — like picking the right direction from a set of options — GPT-5.4 hit 81.9% accuracy. But when asked whether a given claim would actually pan out (feasibility judgment), all models scored between 45% and 52%, essentially random. As for timing, the models were consistently too optimistic: GPT-5.4 was 14 months late, Claude S4.5 was 17 months late, and GPT-4o was a whopping 26 months late. (LLaMA 3.3 fared better, with only a 4-month delay.) On generative solution design, even GPT-5.4's top score of 5.04 out of 10 produced technical roadmaps that didn't align with what actually happened. In other words, your favorite AI can write a plausible-looking research proposal, but it has no idea if the idea will actually work — especially for high-impact, groundbreaking discoveries.