Workflow automation platform Zapier has teamed up with evaluation firm Artificial Analysis to launch AutomationBench-AA, a new independent benchmark for SaaS workflow automation.
The benchmark uses real but anonymized data from Zapier’s platform, designing a private test set with 657 multi-step tasks and 40 simulated SaaS environments — including Gmail, Slack, Salesforce, and Jira — for objective third-party evaluation.
The tough part? It introduces safety compliance guardrails. The AI model must complete its operations without violating any preset enterprise business rules. If it triggers a compliance violation at any point, the task’s final score gets wiped to zero. In the first round, models saw their scores plummet thanks to these penalties; none managed to break 50%.
Among them, Claude Fable 5 — which uses a security downgrade mechanism that falls back to Opus 4.8 for high-difficulty tasks (about 18% of them) — actually completed 73% of the pure task steps, but its overall score was just 48.6%. Gemini 3.5 Flash and GPT-5.5 each finished nearly 70% of steps (68% and 67%, respectively), but their final scores dropped to 42.6% and 42.1%. Open-source leader GLM-5.2 managed only 27.8%.
The results show that how well a model sticks to safety lines during autonomous operations has become a critical challenge for real-world deployment.