Arena's AutoEval AI simulates human voters to rank new models in under an hour

New AI models usually have to wait days for enough human votes to get an Arena leaderboard score. Thanks to AutoEval, that wait just shrank to under an hour.

Arena's new system uses a reward model — trained on millions of real human preference comparisons — to simulate user votes, then calculates an estimated ranking using the platform's existing leaderboard rules. The early score gets labeled as an AutoEval result, and real human votes later correct it. AutoEval currently supports text, vision, image-generation, and code models.

In backtesting, Arena says AutoEval's predicted rankings correlate with eventual human rankings at over 0.98. When two models are separated by more than 10 points, AutoEval picks the winner over 90% of the time; at a 15-point gap, it hits 100%.