Nous Research has published a paper arguing that the tokenizers long relied upon by large language models could one day be replaced. By running controlled tests at the 1.7 billion parameter scale, the team systematically quantified the performance advantages of tokenization and showed that these benefits can be effectively simulated at the byte level using engineering techniques.
The experiments found that simply increasing data throughput and injecting morphological boundaries into a native byte model can dramatically close the performance gap. Under the same compute budget, the simulated compression increased the processing volume per gradient step, directly contributing the largest reduction in validation loss. Meanwhile, overlaying subword boundaries as binary sequences on input bytes gave the model a long-term inductive bias that doesn't leak future information.
While the synergistic effects at larger parameter scales still need to be validated, the tests showed that at the 1.7B scale, the gains from vocabulary parameter scaling and other mechanisms like predicting the next subword are extremely limited. This provides a clear path for developing tokenizer-free models: future architecture improvements should focus on increasing actual throughput and explicitly incorporating morphological priors in a non-leaky manner.