Moonshot AI just published the technical report for its latest large language model, Kimi K3. And it’s a beast of a model: 2.8 trillion total parameters, with 104 billion activated per token. But the parameter count is only part of the story. The company claims that the new architecture, data, and training methods make K3 roughly 2.5 times more efficient to scale than its predecessor, K2. According to Moonshot’s scaling law curve, K3 needs only 40% of the training compute to hit the same validation loss.
So what’s under the hood? Moonshot AI completely rewrote the attention mechanism. K3 uses a custom variant called KDA for handling long sequences, and throws in a global MLA layer every three layers. (The report is still emerging, so the technical details are sparse — but the early numbers are hard to ignore.)