Alibaba is shipping its third-generation GUI agent, and this time the model isn't just staring at your screen — it's doing things. The new foundation model, Qwen-UI-Agent, can see a display, tap buttons, type text, and work across phones, desktops, and web pages. And when a job is better suited for the terminal, it can just run CLI commands instead of making you click through a thousand menus.
Alibaba has been at the GUI agent game for a while. In 2024, Mobile-Agent taught models to read phone screenshots and control taps and swipes. Later efforts like GUI-Owl and MAI-UI started training GUI operations directly into the model and expanded to desktop. Now Qwen-UI-Agent folds phone, desktop, web, DeepSearch, and the command line into a single agent.
The pitch: one agent that can chain together tasks across apps. Look up a high-speed rail schedule, calculate how long it'll take to get to the office, then drop a meeting invite into DingTalk (Alibaba's workplace chat app) — all without you babysitting every step. For data analysis, it can look things up online, crunch the numbers via command line, then generate Excel, PPT, and Word files.
The underlying model is built from Qwen3.5-27B via continued training. On Alibaba's own real-device benchmark, MobileWorld-Real, it scored 92.2% — beating the GPT-5.6 Sol, Claude Opus 4.8, and Gemini 3.1 Pro models tested in the same round.