"Redefine the problem when the geometry breaks Applying diffusion to text failed for years because the mathematics assumed a continuous space — pixels can be smoothly interpolated, but as the speaker explains, 'if you think about text and you take two words then it's not clear what's in between the meaning of two different words.' The team's solution was to redefine 'noise' for a discrete domain: mask out tokens and train the network to predict the missing ones, 'similar in some sense to next token prediction except that things are done out of order.' When a proven method stops working in a new domain, the productive question is not 'how do we force it to fit?' but 'how do we redefine the core concept so it fits naturally?' Source: Arash Vahdat, Latent Space Podcast, Diffusion LLMs with Inception AI"