"Overgenerate then aggressively filter for diversity In building their 'Prismatic Synthesis' dataset, Hashimoto's team found that the standard practice of generating data and keeping it all produces repetitive, homogeneous examples. Their solution was to massively overgenerate candidate data points, use gradient-vector clustering to identify which examples were redundant, and then 'throw out the vast majority of all the data that we just synthesized and only maintain those that are unique and different from each other.' This counterintuitive approach — producing far more than you need, then ruthlessly pruning — beat datasets generated by models 20 times larger. The principle translates directly to any creative or analytical process: generate many options cheaply, then filter hard for genuine novelty rather than keeping everything. Source: Tatsunori Hashimoto, The Cognitive Revolution (or similar Stanford AI podcast), Small Language Models and AI Democratization"