DeepSeek open-sources GPU kernels, revealing two new architecture components
DeepSeek has open-sourced TileKernels under the MIT license, publishing the GPU-level computing code it actually uses for training and inference. GPU kernels are the programs that run directly on the graphics card, setting the speed ceiling for model training and inference. TileKernels are written entirely in Python, relying on the GPU kernel-specific language TileLang to automatically handle low-level optimization — no hand-written CUDA C++ required. DeepSeek says most of the kernels are already close to the hardware's performance limit.
Two brand-new architecture components appear in the code: Engram and Manifold HyperConnection (mHC). Neither has shown up in any published paper from DeepSeek, including V3 and R1. Based on the code structure, Engram is a gating mechanism fused with RMSNorm, complete with forward/backward propagation and gradient reduction — a critical layer in the model backbone. mHC includes Sinkhorn normalization and handles mixed splitting and connections between layers, potentially a more complex information flow mechanism than standard residual connections. Both are packaged as trainable layers, and the code explicitly targets the SM100 (Blackwell) architecture, strongly pointing to a next-generation model DeepSeek hasn't released yet.
The library also covers MoE (Mixture of Experts) routing and gating, various low-precision quantization (FP8, FP4, etc.), batch transposition, and other routine operations. The code can be installed via pip install tile-kernels and requires H100/H200 or Blackwell series GPUs.
These new components come as leaked specs for DeepSeek V4 suggest a 1.6-trillion-parameter model with 384 experts and a 1-million-token context window.