The Qwen team has open-sourced FlashQLA, a high-performance operator library purpose-built for GDN (Gated Delta Network) — the linear attention layer used across the entire Qwen3-Next / 3.5 / 3.6 family. Benchmarks on an H200 show FlashQLA's forward pass runs 2–3x faster than the FLA Triton kernel, with backward pass doubling speed. Against FlashInfer in TP8 (tensor parallelism with 8 GPUs) scenarios, forward pass peaks at 5.33x faster.
https://twitter.com/Alibaba_Qwen/status/2049462758211772663
The core trick lies in exploiting the exponential decay of GDN gate values to do automatic intra-card context parallelism (AutoCP). Traditional methods compute a correction matrix M to stitch sub-sequence states together — a step that can cost more compute than the state matrix itself. FlashQLA discovered that on 60–80% of attention heads, gate values don't stay at 1; only 6–8 chunks of warm-up are needed to push state error below noise, allowing M to be dropped entirely. The system automatically decides whether to enable CP based on batch size, number of heads, and sequence length — no manual tuning required.
At the operator level, FlashQLA doesn't fuse the entire computation into a single kernel, nor does it break it into a series of independent kernels. Instead, it strikes a middle ground: two fused kernels with CP preprocessing inserted between them. Inside each kernel, TileLang is used for warp-level specialization — one producer warp group handles data movement, while three consumer warp groups compute different intermediate variables, overlapping memory access and computation via a ping-pong structure. The speedups are most pronounced in scenarios with many TP shards and few heads — exactly the configuration typical for large-model multi-GPU deployments and on-device agent inference.