Search giant Perplexity AI just open-sourced its production-grade high-performance inference toolkit, pplx-garden. At its heart is a custom Rust peer-to-peer communication library called fabric-lib (aka TransferEngine), designed to break Nvidia's proprietary communication lock-in — letting developers run trillion-parameter models on heterogeneous multi-GPU clusters without buying expensive dedicated network switches.
Traditional distributed LLM inference is brutally tied to Nvidia's high-speed networking, driving up hardware costs and creating supply chain chokepoints. Fabric-lib cuts that hardware dependency: it works perfectly with Nvidia ConnectX-7 cards, but also natively supports Amazon's cheap AWS EFA Ethernet cards, pushing inter-GPU bandwidth to a full 400 Gbps. To get around AWS EFA's physical tendency to deliver packets out of order, Perplexity cooked up a novel ImmCounter synchronization mechanism that enables efficient zero-copy data transfer without assuming packet order. The library also includes a data distribution algorithm tailor-made for mixture-of-experts (MoE) models, overlapping GPU data reception with matrix computation to squeeze every drop of performance out of the decoding phase.
In production, the engineering wins are dramatic. In a disaggregated inference architecture, fabric-lib zips key-value cache between prefill and decoder nodes. During async reinforcement learning training, it syncs and distributes weights for a trillion-parameter model in just 1.3 seconds. And to fix tokenization latency, pplx-garden ships pplx-unigram, a Rust-rewritten tokenizer that slashes CPU consumption by 5–6x, eliminating the tokenization bottleneck for reranking and embedding models.