Meituan's LongCat team, in collaboration with the National University of Singapore, the University of Science and Technology of China, Beijing University of Posts and Telecommunications, and Zhejiang University, has released VitaBench 2.0, a benchmark that shifts the evaluation of large language models from single-turn tasks to long-term, multi-session real-world services. It's designed to test how well AI agents handle personalization and proactive interaction.
The benchmark simulates extended human-AI interactions, featuring 56 user profiles, over 2,000 dynamic preference annotations, and 819 service requests spanning food delivery, in-store dining, and online travel. The key finding: as tool-calling error rates drop significantly, capturing and applying user preferences has become the core bottleneck keeping agents from real-world deployment.
In tests across several frontier models, the weakness in long-term personalization was laid bare. Even in the ideal "full-context" mode — where agents can read all history — the best performer, Claude-Opus-4.6, achieved an average score of just 0.503 across four independent runs (Avg@4), while the top non-reasoning model, DeepSeek-V4-Pro, managed only 0.456.
Most models failed to consistently beat the full-context baseline when given external memory modules. Agent-controlled memories tended to lose information during updates, and retrieval-augmented generation (RAG) was easily thrown off by superficially similar noise. Enabling reasoning mode didn't provide a systematic edge for personalization, suggesting that filtering noise and asking the right questions can't be solved by logic alone.
The tests also exposed a glaring weakness in proactive questioning and preference application. When information was incomplete, models often guessed blindly and rushed decisions. Claude's agent dropped from 46.0 to 27.4 on proactive interaction tasks, DeepSeek fell from 44.1 to 27.8, and GLM tumbled from 36.4 to 19.3. Even when the most accurate preference data was directly injected into memory, DeepSeek and Claude scored only 52.7 and 51.2, showing that prioritizing, integrating, and acting on preferences is far harder than merely extracting them.
Error analysis shows that as model capabilities improve, tool-use errors are shrinking, while errors from preference misunderstanding are taking over. The next big barrier for AI agents isn't better tool calling — it's learning to actually hear what users mean between the lines.