MiniMax's M3 sparse attention claims 9.7x speedup on million-token context

MiniMax engineering lead Skyler Miao teased the upcoming M3 model on X, revealing a new attention architecture called MiniMax Sparse Attention (MSA).

For contexts up to 1 million tokens, MSA claims a 9.7x speedup in the prefill stage and a 15.6x speedup in decode compared to the dense attention used in M2.

As context windows push toward a million tokens, the bottleneck is shifting. VRAM used to be the main constraint, but beyond 100k tokens, the sheer computational cost of dense attention becomes the primary obstacle for local deployment. Redis creator Salvatore Sanfilippo (antirez) called MSA "the right path," arguing that dense attention overhead is unsustainable for local inference. This aligns with industry trends: DeepSeek V4's optimizations for 1M context also rely on a mixture of attention mechanisms, including compressed KV, compressed sparse attention, high-compression-rate attention (128x compression before dense attention), and sliding window attention, balancing KV cache pressure and compute savings.

This isn't MiniMax's first pivot on attention. The M1 series leaned heavily on linear attention like Lightning Attention, but M2 reverted to full attention. In an October 29, 2025 blog post, pre-training lead Sun Haohai detailed the trade-offs: linear or sparse attention performs well on standard benchmarks but can degrade on code, math, agent tasks, and long-chain reasoning. At that time, supporting infrastructure like prefix caching, low-precision state storage, and speculative decoding was also immature. MSA signals a renewed push toward efficient attention.

The design path is clear — block-wise indexing and sparse computation — but whether MSA can deliver without compromising reasoning quality remains to be seen, pending the full technical report, open-weight release, and third-party replication.