Zhipu AI has rolled out a turbocharged version of its GLM-5.1 API for select enterprise customers, claiming output speeds of 400 tokens per second — a new global record for end-to-end inference from a major model provider's official API.
The high-speed variant retains the full capabilities of the flagship model. Under the hood, it's powered by a custom inference engine co-developed with the TileRT team. This engine totally rethinks GPU scheduling: at compile time, the model is statically compiled into a persistent engine kernel that lives on the GPU. During single-card inference, computation, asynchronous I/O, and communication are all broken down into tile-level micro-tasks and launched with a single kernel call. Intermediate results between operators are passed directly via registers and shared cache, eliminating the latency bubbles caused by frequent kernel launches and memory reads/writes in traditional inference.
When scaling to multiple cards, TileRT extends the specialization strategy across an 8-card NVL topology, transforming what were once homogeneous GPU nodes into heterogeneous workers with distinct roles. For GLM-5.1's attention layer, GPU 0 is assigned as a sparse indexing worker — handling sparse index construction and routing decisions — while GPUs 1 through 7 run as MLA workers, handling compute-heavy stages and pushing all communication down into the tile-level task pipeline. This achieves deep overlap between computation and cross-card communication.
The high-speed service is now available to select enterprise customers on Zhipu's MaaS platform. Going forward, the company plans to further optimize FP8 inference and ultra-long-context production environments, offering more deterministic performance for latency-sensitive applications like AI coding, real-time interaction, and live voice.