"Iterate at medium scale before committing to large Asked about the path to larger diffusion models, the speaker was explicit about sequencing: 'a lot of the time we still have to reinvent new things and it doesn't make sense to do all the R&D at the largest possible scale. We can iterate much more quickly if we try out our ideas at small to medium scale, just because iteration is faster.' Every 10x increase in data or parameters 'comes with a lot of engineering challenges and it often means you have to change a lot of the infrastructure.' Building or scaling anything — a team, a product, a model — benefits from proving the approach works cheaply before making the expensive bet. Source: Arash Vahdat, Latent Space Podcast, Diffusion LLMs with Inception AI"