See the OpenAI blog post and their paper.

Contrary to conventional wisdom, we find that the performance of CNNs, ResNets, and transformers is non-monotonic: it first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction.