AMD's MI355X challenges Nvidia B200 in long-context AI inference

AI cloud provider Zyphra has published the first end-to-end inference benchmark for AMD's flagship MI355X chip, pitting it directly against Nvidia's B200 in real-world single-node tests running DeepSeek V3.2, Kimi K2.6, and GLM-5.1.

The results highlight the fundamental hardware difference: AMD's killer feature is 288GB of HBM memory, dwarfing B200's 180GB. That lets a single MI355X cache massive contexts without splitting across cards, saving hardware cost. But AMD trails in chip-to-chip interconnect. B200 uses NVLink switches to hit 900GB/s between any two cards; MI355X relies on point-to-point links, making multi-card communication less efficient.

To compensate, Zyphra developed Tensor Sequence Parallelism (TSP) and Tree Attention — algorithms that replace traditional ring networks with tree-based communication, overlapping computation and data transfer in the decode phase to work around AMD's interconnect limitations.

The final scores are clear: Nvidia B200 still dominates per-request speed. But as context length grows, Zyphra's inference stack pushes MI355X throughput close to B200's. The takeaway: with proper software optimization, AMD's memory advantage can let it compete head-to-head with Nvidia's best in long-context production environments.

Next, Zyphra plans to use this architecture to support DeepSeek V4 Pro (1.6 trillion parameters) with a 100-million-token context. The team will also develop custom low-precision quantization for MI355X and introduce diffusion speculative sampling to wring more performance from the chip.