Large language models face a persistent problem: once deployed, they can't learn from new conversations. Current fixes focus on expanding context windows and improving retrieval speeds, but that only gives models temporary access to information within a single session. Close the chat window, and the knowledge is gone. The real bottleneck isn't retrieval speed — it's physically rewriting the model's underlying weight parameters based on what it learns from interactions.
Online Policy Self-Distillation (OPSD) offers a new path. When an LLM encounters a task, its "teacher state" — which holds the full long-term context — generates a high-quality answer. Then, via backpropagation, the system computes the token-level probability difference between the base model (the student) and the teacher state, providing a dense supervision signal that pushes the student toward the smarter, high-scoring state.
Unlike supervised fine-tuning (SFT), which forces the model to memorize every word of dialogue, self-distillation extracts only the decision-making experience needed to maintain performance. This extremely sparse parameter update avoids catastrophic forgetting, preserving the model's general knowledge.
Another, more forward-looking approach is dreaming simulation. When faced with a complex task, the model consumes massive inference-time compute to self-play inside its own head. It automatically builds a virtual simulator based on observed patterns, then runs tens of thousands of trial runs. If a trial succeeds, the system logs the successful trajectory as a lesson and updates the base model's weights. Compared to lightweight compression that just generates brief summaries, dreaming simulation burns huge amounts of compute in the cloud for repeated rehearsal — it's considered the fourth dimension of LLM scaling.
By 2027 or 2028, AI agents could undergo performance reviews after a week of collaborative work with humans. If they pass, the system could use OPSD or dreaming simulation to distill that week's real-world experience into the model's underlying weights — letting LLMs truly get smarter the more they're used.