Xiaomi's MiMo UltraSpeed mode hits 1,000+ tokens per second on a trillion-parameter model

Xiaomi's MiMo team, in collaboration with its AI compiler optimization group TileRT, has unveiled the MiMo-V2.5-Pro-UltraSpeed inference mode. On a single standard 8-GPU node, they've achieved over 1,000 tokens per second — peaking at around 1,200 tokens/s — on a 1-trillion-parameter mixture-of-experts (MoE) model. That's the first time anyone's broken the kilotoken-per-second barrier on a trillion-parameter model using only off-the-shelf hardware and model-system co-design, no exotic wafer-scale or on-chip SRAM ASICs required.

The speed boost comes from a tight integration of model architecture and system optimization. On the architecture side, MiMo-V2.5-Pro uses hybrid sliding-window attention (Hybrid SWA): only 10 of its 70 layers use full attention; the other 60 use local window attention with a window size of 128. That slashes KV cache memory requirements to about 1/7 of a conventional full-attention design. For quantization, the team applied selective 4-bit quantization to the MoE expert layers using the OCP Alliance's MX micro-scaling format, cutting memory footprint and memory bandwidth pressure, while keeping FP8 precision for the core non-expert modules to maintain output quality.

On speculative decoding, the DFlash framework uses a lightweight block-diffusion model and non-causal attention masks to break the usual token-by-token autoregressive bottleneck. It predicts whole blocks of candidate tokens in a single forward pass, boosting GPU utilization and verification efficiency.

Under the hood, the TileRT inference engine provides a closely matched compiler and custom compute kernels. It runs in persistent execution mode, keeping the compute pipeline resident on the GPU to avoid kernel launch overhead and pipeline bubbles.

Xiaomi MiMo has open-sourced the 1T model weights on Hugging Face, including FP4 and DFlash speculative decoding checkpoints. The UltraSpeed API is available for limited trial applications from June 9 to June 23 (PDT). After the trial, pricing for the UltraSpeed API will be set at 3x the standard version — but it's nearly 10x faster. The web-based chat interface is free for a limited time. Enterprise and large-scale inquiries can reach out to [email protected].