Moonshot AI just dropped PerceptionBench, a benchmark designed to catch when multimodal models fail because they literally didn't see the image.
Most existing evaluations mix vision, knowledge, and reasoning into one messy pile. When a model gets something wrong, you have no idea if it misread the picture or just thought wrong.
PerceptionBench starts by collecting failure cases from top models across 42 benchmarks, then reverse-engineers them into 10 fundamental visual abilities — counting, localization, text recognition, spatial judgment, and so on.
The team built 3,000 questions from those findings. Each question tests exactly one ability: look at the image, answer the question. No reasoning required. No external knowledge needed.
Code and dataset are already open.