I’ve recently started looking at Deep Learning methods (specifically LSTM) for doing some time series analysis. While I understand the concepts involved in the network in isolation, one thing that confuses me is that how do we derive the architectures that work for a task.

I did some research into this and an answer that usually comes up is to dig into the literature and see what architect similar experiments used and start from there. My question is that is there a qualitative metric or a method that guides us for creating optimal architectures without using the layers as Lego bricks and connecting them hoping to get the best results.