Pretraining data for AI models has traditionally relied on a mix of web crawls and select sources, like digitized books. This approach is shifting, highlighted by emerging models that utilize synthetic datasets. In 2025, several major models, including Minimax and Trinity, adopted extensive synthetic datasets in their training. Pleias even experimented with fully synthetic training using a model called Baguettotron, trained in an entirely synthetic environment. The core idea behind synthetic pretraining is that the data typically collected doesn't effectively produce the desired model capabilities, prompting a need to rethink data design from the start of the training process. Synthetic pretraining entails integrating synthetic data throughout the training cycle, rather than just at the end. This approach requires early involvement of data design teams and continuous monitoring of specific capabilities during training. Zeyuan Allen-Zhu describes this as creating a "synthetic playground," where controlled environments allow for precise experimentation. The initial use of synthetic data can enhance both data and model efficiency. For instance, Microsoft's Phi 1.5 model, released in 2023, was the first to be trained entirely on synthetic data, achieving results comparable to much larger models trained on traditional datasets. However, the initial enthusiasm for synthetic pretraining faced challenges. After the release of Phi 1.5, the research teams reverted to using a mix of preexisting data and synthetic augmentation. They found that while synthetic pretraining offered a promising avenue, it required a complete overhaul of the training environment. The complexity of ensuring that synthetic data encompasses the necessary capabilities often led researchers back to more conventional methods. As of early 2026, there’s an evolving landscape of reusable synthetic datasets, with models like Nemotron-Synth and SYNTH gaining traction. The ongoing debate highlights the need for a deeper understanding of synthetic data techniques, which vary widely in effectiveness and relationship to existing data sources.