AIDwarkesh Podcast1h ago
Analysis: from 2019 to 2025, gains in pretraining compute efficiency
Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements
TL;DRAI got better mostly by using better data, not smarter model designs.
Why it matters: Understanding where gains come from shapes how companies allocate R&D resources and compute budgets going forward.
Breaking down 6 years of pretraining progress into data vs model improvements — How much of the rapid progress in AI that we've seen over the last few years 1 …
Read full articleSource: Dwarkesh Podcast · Opens in new tab