Moonshot AI has released the technical report for Kimi K3, its latest large language model. K3 comes with a staggering 2.8 trillion total parameters, activating 1040 billion per token, and a completely new base architecture compared to its predecessor K2.
But the bigger story isn't just about raw parameter count. Moonshot AI says the new architecture, data, and training methods give K3 roughly 2.5x better scaling efficiency than K2. According to their scaling law curve, reaching the same validation loss now requires only about 40% of the training compute.
K3 starts by rewriting attention. It uses KDA (key-value compressed attention) for long sequences, with a global MLA (multi-head latent attention) layer every three layers. KDA compresses previous context into a fixed-size state, slashing compute and cache pressure for million-token contexts, while MLA handles global information exchange.
It also introduces attention residuals. Traditional models just stack information from all previous layers; K3 lets each layer pick and choose from earlier network blocks. That way, as the model gets deeper, early information doesn't get diluted.
The mixture-of-experts (MoE) component gets a redesign too. K3 has 896 routed experts, activating 16 per token — double K2's count. Routed experts first compute in a compressed space before returning to the main model. New activation functions and load-balancing algorithms keep training of this massive model from going off the rails.
What really boosts agent capabilities is the post-training. Moonshot AI trains separate general, agent, and code models, then trains three thinking intensities for each direction, and finally merges the nine experts. Training tasks can involve thousands of tool calls, persistently retaining file, app, and virtual machine states.
K3 has already matched or beaten top closed-source models like Fable 5 and GPT-5.6 Sol in several code, search, and tool-use benchmarks. Its leap isn't just from throwing more parameters at the problem — it's from pushing a larger base, new architecture, and million-token agent reinforcement learning to their limits.