I was wondering about how people come up with new architectures for neural nets. I am currently working on an object detection problem, and the models that are SOTA are huge.

With models that have more than 500 layers and millions of parameters, how does one come up with an architecture that is better than the existing ones? I know that papers are mostly written around innovative ‘concept’ blocks (like skip connections or a feature pyramid block), so are researchers just iterating on all possible combinations of blocks to come up with an answer?