Doubled training data to 33T is why DeepSeek V4 is late, says DeepMind researcher

DeepSeek V4's technical report reveals that V4-Flash and V4-Pro were pre-trained on 32T and 33T tokens, respectively — roughly double the ~15T tokens used for V3. The report admits to "significant instability challenges" during training, with repeated loss spikes traced back to outliers in the MoE layers. The routing mechanism itself exacerbates these outliers, and simple rollbacks weren't enough.

DeepSeek implemented two workarounds in practice: Anticipatory Routing, which decouples routing index computation from the backbone network update and only triggers when a loss spike is detected (at about a 20% overhead), and SwiGLU Clamping, which clips activation values to a fixed range to suppress outliers directly. The report claims both are effective but concedes that "the underlying principles are not yet fully understood."

The delay has been a topic of speculation; leaked specifications claim the model packs 1.6 trillion parameters and 384 experts.

Google DeepMind researcher Susan Zhang (formerly at Meta AI and OpenAI) commented that the instability from doubling the training data "explains the delay." She described the two fixes as "band-aids," but acknowledged DeepSeek's technical transparency.