Tencent just dropped DiffSpot on Hugging Face — a benchmark designed to test whether multimodal AI models can spot tiny visual changes in web pages. This isn't your typical "spot the difference" puzzle. Instead, DiffSpot tweaks a single CSS property of one element in an HTML page and asks the model to pinpoint what changed.
The dataset includes 4,400 pairs of web screenshots: 3,900 with real changes spanning 13 CSS modification types across three difficulty levels, plus 500 identical pairs to catch false positives. Every sample is carefully filtered so that pixel changes land strictly inside the target element — no noisy edge cases.
The results? Not great. In zero-shot tests across 13 cutting-edge vision-language models (VLMs), the best performer — Gemini 3.1 Pro — scored only 47.2% overall accuracy, with a recall of 40.7% on real changes. That means it missed roughly 60% of alterations. On the hardest tasks, every model's recall dipped below 23%.
Among open-source models, Kimi K2.5 led with 42.2% overall accuracy, beating GPT-5.4's 38.3% and Claude Opus 4.7's 38.9%. Qwen3.5-VL-397B came in second among open-source entries at 37.6%.
DiffSpot also busts a common assumption: bigger visual change doesn't always mean easier detection. Across different CSS properties, neither pixel-level change magnitude nor CLIP feature distance reliably predicted model recall. In other words, models miss changes not just because they're small, but because they fail to truly understand web elements, styling properties, and localized visual shifts.
Some models swing the other way — becoming overly conservative to avoid false alarms. Qwen3-VL-235B-Instruct posted zero false positives on the 500 unchanged pairs, but its recall on real changes was a dismal 5.1%. It's basically defaulting to "nothing changed" rather than actively hunting for subtle differences.