For months, Zhipu AI users were getting gibberish output from GLM-5 — scrambled characters, repeated phrases, obscure Chinese characters — but only when they used it for coding. Turns out the cause was two silent race conditions hiding deep in the KV cache.
Zhipu published a postmortem detailing what went wrong with its GLM-5 series models in Coding Agent scenarios. Problems started popping up in March, but only under high concurrency and long contexts (averaging over 70K tokens) in coding agent tasks. Under standard inference conditions, the issue was impossible to reproduce. Zhipu says its inference system handles hundreds of millions of Coding Agent calls per day. The company recently extended its triple-usage promotion for the GLM Coding Plan through June 30.
After weeks of investigation, the team pinned down two independent race condition bugs at the infrastructure level.
The first lived in the prefill-decode separation architecture (where prefill and decode run on different nodes). When the decode side timed out and cancelled a request, it freed the KV cache (a buffer that stores computed attention states to avoid recomputation). But the prefill side's RDMA write hadn't finished yet. A new request got assigned the same GPU memory block, and the old data overwrote the new. The fix: add explicit synchronization before freeing memory, ensuring the write completes before release. After the fix, the anomaly rate dropped from around 0.15% to below 0.03%.
The second bug was in HiCache (multi-level KV cache). During asynchronous swaps from CPU memory to cache, there was no synchronization point between the load pipeline and the compute pipeline. The compute side could start reading before the data finished loading. After the fix, this class of anomaly disappeared entirely, and the patch was submitted to the SGLang community (PR #22811).
During the investigation, the team made an unexpected discovery: the acceptance rate of speculative decoding (a speed-up trick where a small model drafts tokens and a big model verifies them) could serve as an anomaly detection signal. When the output was garbled, almost all draft tokens were rejected; when it was repeating, the acceptance rate was abnormally high. The team built online monitoring that automatically aborts generation and retries when the rate crosses a threshold.
After fixing the bugs, the team also optimized a bottleneck: LayerSplit KV Cache stores only a subset of layers' KV cache per GPU, using broadcast for collaborative computation. At a 90% cache hit rate, for request lengths between 40K and 120K tokens, throughput improved by 10% to 132%, with longer contexts seeing bigger gains.
