Adobe Research, the Australian National University (ANU), and New York University (NYU)'s team led by Saining Xie have open sourced the code, model weights, and training data for RAEv2 (Representation Autoencoders v2). This second-generation diffusion model for image reconstruction—positioned as a replacement for traditional variational autoencoders (VAEs)—fixes the original's abysmal reconstruction quality, inability to use standard classifier-free guidance (CFG), and painfully slow convergence. On ImageNet, RAEv2 achieves a global FID (gFID) of 1.06 in just 80 training epochs, a 10x improvement in convergence speed.
The team made three key architectural upgrades. First, instead of using only the last layer of a pretrained encoder like the original, the new architecture employs a "multi-layer representation" scheme that directly sums the outputs of the last K encoder layers. In ultra-high-dimensional spaces, this simple addition preserves the structure of the underlying subspaces, letting users dial in the optimal Pareto frontier between reconstruction quality and compression efficiency simply by tuning K. Second, the team clarified how the representation autoencoder and representation alignment (REPA) complement each other. When used together, the autoencoder mainly captures global semantic info, while REPA provides a strong constraint on spatial details and structure. This complementary design lets stronger, spatially-aware encoders like DINOv3 shine in generation tasks—in the original model, DINOv3 actually performed worse than DINOv2.
The most critical insight: REPA essentially performs x-prediction in the autoencoder's representation space. The original autoencoder struggled with standard CFG and required training a separate auxiliary diffusion model (AutoGuidance). The new architecture cleverly reinterprets the diffusion model's output as an x-prediction in representation space, allowing the alignment head to act as a natural weak guidance baseline. This delivers "free" internal guidance without any extra training or inference cost.
In evaluation, RAEv2 achieved better gFID, FDr6, and Monge Distance scores across six representation spaces. The team also introduced a new efficiency metric: EP_FID@k (epochs required to reach a target gFID). To hit a gFID below 2, the original model needed 177 epochs, while the new model needs just 35. In downstream tasks like text-to-image generation and embodied navigation world models, RAEv2 showed strong generalization and effectively eliminated the inter-frame flickering that plagues video synthesis.