"Validate quality by checking consistency across attempts When generating fully synthetic math problems and solutions where no ground truth exists, Hashimoto's team needed a way to verify correctness without human review. Their solution: 'ask the model to solve the same problem multiple times and then check whether the final answer is identical to each other — if not then we worry that the quality might be bad.' He calls it 'a very crude way of filtering data, but it worked well enough' and describes it as 'a powerful method for controlling the quality of synthetic data without human validation.' This consistency-as-proxy-for-correctness heuristic is immediately applicable whenever you need to assess the reliability of any AI output, analysis, or decision in the absence of a definitive external check. Source: Tatsunori Hashimoto, The Cognitive Revolution (or similar Stanford AI podcast), Small Language Models and AI Democratization"