DeepSeek drops V4 preview

DeepSeek has open-sourced a preview of its V4 series under the MIT license, with weights already live on Hugging Face and ModelScope. The lineup includes two MoE models: V4-Pro with 1.6 trillion total parameters and 49 billion activated per token, and V4-Flash with 284 billion total and 13 billion activated. Both support a 1 million token context window.

Three architectural upgrades power the series. A hybrid attention mechanism — combining compressed sparse attention (CSA) and heavily compressed attention (HCA) — dramatically cuts long-context costs. At 1M context, V4-Pro's per-token inference FLOPs are just 27% of V3.2's, and its KV cache (the memory used to store historical info during inference) is only 10% of V3.2's. A manifold-constrained hyperconnection (mHC) replaces traditional residual connections to improve cross-layer signal stability. Training now uses the Muon optimizer for faster convergence. Pretraining data exceeds 32 trillion tokens.

Post-training happens in two stages: first, domain experts are trained separately using SFT and GRPO reinforcement learning; then, online distillation merges them into a single unified model. V4-Pro-Max, the highest-inference-effort mode, claims to be the strongest open-source model available today, with coding benchmarks at the top tier and significantly narrowed gaps to closed-source frontiers on reasoning and agent tasks. V4-Flash-Max, given enough thinking budget, approaches Pro's reasoning performance, though it's limited by parameter count on pure knowledge and complex agent tasks. Weights are stored in FP4+FP8 mixed precision.