A key finding in large model post-training: "same-trajectory sampling" — training a model on its own real-time attempt paths — is critical for preventing degradation and improving problem-solving ability. Online reinforcement learning (RL) and on-policy distillation (OPD) outperform traditional supervised fine-tuning (SFT) precisely because they let the model optimize based on its own written steps rather than memorizing external standard answers.
SFT forcibly imprints standard answers, applying uniform adjustment pressure to every token, which easily disrupts the model's existing knowledge structure and causes forgetting. In contrast, RL and OPD let the model find and reinforce the best steps within its own drafts. This not only avoids the cumulative error of "one wrong word at the start, everything derails," but also ensures updates only occur within the model's known knowledge regions, maximally preserving native capabilities.
In a "minimal code editing" experiment, whether using an SFT or RL teacher for on-policy distillation, the student model achieved pass@1 rates of 80.0% and 78.7% respectively — both surpassing the teacher model. Even when the SFT teacher was severely degraded by over-fine-tuning (dropping from 0.320 to 0.286 on the LiveCodeBench coding test), its student still scored 0.297, almost unaffected by the teacher's defects, proving that same-trajectory practice effectively filters out bad teacher habits.
Currently, DeepSeek-V4 and GLM-5 have introduced on-policy distillation to merge expert model capabilities. In expert training, domains with clear right/wrong answers like coding and math are better suited for RL, while creative and knowledge-based subjective tasks are more appropriate for on-policy distillation. The ultimate fine-tuning algorithm of the future must find a new mechanism within the same-trajectory training framework that combines distillation's high efficiency (high information density) with RL's objectivity (unbiased updates).