"Let autonomous systems find what expert intuition misses Karpathy had spent two decades hand-tuning neural network training runs and believed a particular model was already well-optimized. He then let an autonomous agent run overnight and it surfaced improvements he had overlooked: "it came back with tunings that I didn't see — the weight decay on the value embeddings and my adam betas were not sufficiently tuned." The lesson is that deep expertise creates confidence that can blind you to interactions between variables, and handing the search to an automated system that evaluates objectively can outperform even years of human intuition. Source: Andrej Karpathy, Conviction, Episode with Andrej Karpathy"